Self Diagnosis from Common CTSTATUS Messages
The c-treeACE database engine is designed at every level to require as little attendance and maintenance as possible. Even for those times when the server detects potential problems, every attempt is made to output the cause, and possible solution. The CTSTATUS.FCS log file retains a record of any notable information and should be inspected from time to time to prevent and diagnose most common issues that can arise.
Check your CTSTATUS.FCS file any time you have cause to believe there is a problem with the database engine. This status log file can be found in the current working directory of the c-treeACE process. Typically, this is the directory of the server process binary, or an area pointed to by the LOCAL_DIRECTORY configuration keyword.
Here is a reference list of the most common messages your FairCom support team will ask about or look for in your status logs should you have problems. In many cases, simply searching for these is enough to handle most situations.
Up and Running
Upon server startup, a handful of messages are written to the status log. Usually, this is a quick activity. Sometimes it may take a while if auto-recovery from a previous abnormal shutdown is taking place. Be sure the "Server is Operational" message is output if you are unable to connect to the server. Once this message is output, you are ready to go.
Fri Aug 13 10:30:36 2010
- User# 00001 c-tree(R) V9.3.27711 Server Is Operational -SN 50475612
Fri Aug 13 10:30:36 2010
- User# 00012
Server using TCP/IP Socket Port Number: 5597
Any startup errors will be noted in this output should the server fail to start. Server Start Up Errors
Note: The server serial number and TCP/IP connection port is output in this final information for your reference.
Build Info
When requesting server support from your FairCom team, one of the first questions you will be asked is "What is your server build?". This information is output in the status log output on server startup. Be sure to identify your version and build information and provide this when you make a support request.
Fri Aug 13 10:30:34 2010
- User# 00001 Startup c-tree(R) Server - V9.3.27711(Build-100526)
Server Name and Ports
Clients connect to the server via a communications port. Usually TCP/IP is used across networks. (Windows clients and servers on the same machine will use a shared memory protocol by default.)
Fri Aug 13 10:30:34 2010
- User# 00001 Server Name: FAIRCOMS
Fri Aug 13 10:30:36 2010
- User# 00001 Server using TCP/IP SQL Socket Port Number: 6597
Your ISAM port is identified by the server name. There are two connection ports that are possible with c-treeACE SQL. One is the ISAM port, and the other is the SQL port; the SQL port is numeric only. These ports are configured in your ctsrvr.cfg configuration file.
Note: The ISAM TCP/IP port number is determined by the following formula. Start at base 5000 (configurable) and add the ASCII sum of the server name characters. As a result, keep in mind two different server names could potentially result in the same port number calculation! (For example, PROD_SERVER1 = SERVER_PROD1)
VDP (Communication) Errors
A very common question our support team is asked is "What are these 127 and 128 errors?". Communications Errors (127/128)
Tue Oct 05 16:34:55 2010
- User# 00017 ctntio: read error - O17 bytes=0 pErr=127
|ADMIN||: 161
- 127 ARQS_ERR Could not send request
- 128 ARSP_ERR Could not receive answer
In most cases, this error is an informational message that a connection to the server, usually a dropped client, has occurred. When a client establishes a connection to the server, that connection remains through the life of the client until the client issues a StopUser() or CloseISAM() call. FairCom Client/Server Communication
Consider the case where a client application abnormally terminates though. A proper disconnection does not take place. The connection from the server's view appears valid, however, until the server issues a read or write operation on the connection. Only at this time will the server note that the connection is no longer available, and then outputs a message to CTSTATUS.FCS indicating so. A ARQS_ERR (127) indicates that the server was attempting to write information to the connection and the ARSP_ERR (128) indicates the server was attempting to read information from the connection.
Normally, a few of these messages is not an issue and can be safely ignored. An easy example to generate such a message is to connect with a c-treeACE command line client utility and use control-C to break out of the application. You will frequently find a message logged after this action.
Sometimes a very large number of these messages appear in the status log concerning administrators. While these errors can be masked from CTSTATUS.FCS output with the CTSTATUS_MASK VDP_ERR configuration keyword, it is recommended that you make every attempt to identify the root cause of the disconnections. A status log with a large number of these messages, in nearly every case, is indicative of an application that failed to properly disconnect from the server upon closing. Many times, it is simply that the application was closed improperly, and these can be safely ignored. In some multithreaded applications, tracking of connections to the server is miscalculated, and this is a symptom of a larger problem that should be inspected. This message has also indicated a faulty networking environment in several instances due to dropped connections. Faulty network interface cards, cables, switches and even routers have been identified after the appearance of these messages.
FYI -- VDP originally designated "Virtual Device Protocol".
Note: You may see many of these errors appear all at once if the server is stopped while clients are connected. This is normal server end processing.
License Notice
c-treeACE V9 introduced the ability to limit the number of CPUs the server was licensed, or configured to run on. This message will be output every five (5) minutes when the activation key specifies fewer CPUs than detected on the system. Contact your FairCom business representative to arrange for a licensing change, or include the CPU_AFFINITY keyword on selected platforms to limit the number of CPUs the c-treeACE process can access.
Mon Aug 23 23:12:26 2010
- User# 00008 LICENSE NOTICE: * * * * * * * * * * * * * * * * * * * * * *
c-treeACE is licensed for 2 CPU's, but 4 CPU's have been detected in the
host machine. Either upgrade the c-treeACE license to support a greater
number of CPU's or bind c-treeACE to specific CPU's using the
CPU_AFFINITY configuration keyword.
Note: On Windows machines, when run as a tool tray process, a "balloon tip" will appear every five minutes, along with this message in the status log.
Routine Diagnostic Messages
This following message can occur during normal server startup. I000000x.FCS files are server housekeeping files created at server startup, and deleted upon a clean server shutdown. If the file is missing during startup (normal) the check for the file fails, and this message is output. During startup, low level I/O errors are detected and output as part of normal startup operations. These can be safely ignored
Fri Aug 13 10:30:36 2010
- User# 00001 idltfil: File <./data/\I0000000.FCS> unlink failed with error={2}
Dynamic Dump
The dynamic dump is the preferred backup method for online c-tree files. When processing a dynamic dump, a series of messages reporting progress through the dump is output. It is important to inspect for errors within these messages to ensure you have a pristine backup of your data.
Sat Jul 18 04:45:00 2009
- User# 00011 DD: processing script file...
Sat Jul 18 04:45:00 2009
- User# 00011 My_DynDump.scr
Sat Jul 18 04:45:00 2009
- User# 00011 DD: Begin dynamic dump...
Sat Jul 18 04:45:00 2009
- User# 00011 My_DynDump.scr
Sat Jul 18 04:45:02 2009
- User# 00011 DD: Effective Time
Sat Jul 18 04:45:02 2009
- User# 00011 DD: Dynamic Dump begins with log #66
Sat Jul 18 04:45:02 2009
- User# 00011 DD: could not access...: 12
Sat Jul 18 04:45:02 2009
- User# 00011 /u/My_DumpNow.scr: 13
The most frequent errors are files that could not be found. For example, the dump script was not updated with file name changes. A missing or deleted dump script file will also cause errors.
Tip: Configure your dump script files as read only to prevent them from accidental deletion.
Automatic Recovery
During startup, the server examines its transaction logs to determine if there was a previous clean shutdown. The transaction logs maintain changes to transaction controlled data and index files and are used to recover information to bring these files back to a consistent state. Various messages appear during the multiple stages of recovery.
Mon Nov 23 10:46:26 2009
- User# 00001 Index repair time: 0 second(s) for 1 repair(s).
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Recomposing index file FAIRCOM.FCS DI:
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Processing abort node list entries.
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Recomposing index file mark.idx:
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Processing LOGIDX node entries.
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Checking index delete stack.
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Recomposing index file mark.idx M#01:
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Processing LOGIDX node entries.
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Recomposing index file mark.idx M#02:
Mon Nov 23 10:46:26 2009
- User# 00001 tranrcv: Processing LOGIDX node entries.
Mon Nov 23 10:46:26 2009
- User# 00001 Index composition time: 0 second(s).
Some files, however, may no longer be present. The SKIP_MISSING_FILES configuration option is used to skip files that may no longer be present, yet referenced in the server's transaction logs.
Tue Nov 04 16:15:38 2009
- User# 00002 RCVchk: skipped .\ctreeSQL.dbs\customer.dat
Tue Nov 04 16:15:38 2009
- User# 00002 .\ctreeSQL.dbs\inventory.dat
Tip: Add SKIP_MISSING_FILES to your ctsrvr.cfg configuration file, however, comment it by default. This way it is immediately available should you really need to use it during a server recovery process. Only use this option when you are certain files should actually be missing, otherwise they will be lost after recovery. In that case, you will only be able to restore them from a prior backup.
Clean Shutdown
Once the Server shutdown completed message appears, the server is safely down, and any maintenance or update activities can take place.
Fri Aug 13 10:51:02 2010
- User# 00013 Server shutdown initiated
Fri Aug 13 10:51:04 2010
- User# 00013 Communications terminated
Fri Aug 13 10:51:04 2010
- User# 00013 Perform system checkpoint
Fri Aug 13 10:51:04 2010
- User# 00013 Server shutdown completed
Fri Aug 13 10:51:04 2010
- User# 00013 Maximum memory used was 21584854 bytes
Fri Aug 13 10:51:04 2010
- User# 00013 Maximum number of lock hash bins for a file was 128
Fri Aug 13 10:51:04 2010
- User# 00013 Maximum number of lock hash bins for a user was 256
Fri Aug 13 10:51:04 2010
- User# 00013 Maximum number of preimg hash bins for a user was 512
Failed Server Process
Oops. Your c-treeACE server fails for some reason. While it is not possible to detect every situation which may require the server to shut down, a best attempt is made. By far, the most common cause is lack of disk space for either files, or the server's transaction logs. Without disk space, the server cannot continue operation, and will immediately come down as clean as possible to protect your data. This is noted with a READ_ERR (36) or WRITE_ERR (37) in the final uerr_cod.
Thu Aug 30 12:58:55 2007
- User# 00006 Dumped stack for server process 2308, log=1, loc=60, rc=0
Thu Aug 30 12:58:56 2007
- User# 00006 O6 M2 L60 F279 P47c000x (recur #1) (uerr_cod=37)
Another very common situation is that the server was already started, and a second attempt was made.
Tue Oct 12 16:25:18 2010
- User# 00001 Process ID: 1484
Tue Oct 12 16:25:18 2010
- User# 00001 Is another server running in this workspace?
Tue Oct 12 16:25:18 2010
- User# 00001 Dumped stack for server process 1484, log=1, loc=54, rc=0
Tue Oct 12 16:25:18 2010
- User# 00001 O1 M99 L54 F537 P1x (recur #1) (uerr_cod=0)
Be sure to never start a server that is undergoing auto-recovery. It is always recommended to check the status log BEFORE you attempt to restart a server you think has not started. Look for the "Server shutdown completed" message, or output such as the previous before restarting any server.
On many platforms, a core or dump file will be generated. Be sure to save off this file should your FairCom support team request this information. The stack trace and core file information will in many cases determine an immediate cause and solution to the problem.
Concluding Remarks
This certainly doesn't cover every message that can occur, however, with an occasional routine inspection of your CTSTATUS.FCS log, you should be able to identify potential problems before they happen. Even in those cases where a failure occurs, there may be enough information to quickly diagnose the situation and get back up and running quickly. And always remember, your FairCom support team is here ready to help.
See Also
|
