Helpful Debugging Tips

Inevitably with complex systems, bugs always seem to creep in. As such, our developers sometimes have to find and analyze difficult-to-trace bugs. In a separate article in this issue of the eNewsletter, we describe how to use the loadable c-treeACE database engine. Debugging applications that depend on shared libraries can be especially tricky. Here are some tips and tricks we commonly use here at FairCom for debugging in general.

Some of our most common tools are dbx, gdb and Microsoft Visual Studio. It is also valuable to become familiar with all of the tools available on a particular platform. ldd, strace, dtrace, and other such tools prove invaluable at times.

Custom Breakpoints

On Windows, FairCom engineers developed an internal breakpoint function which detects when a debugger is attached to the c-tree Server process, and if so, it breaks into the debugger. Much like an assert() statement, this debug breakpoint function is called at interesting points in the code when the code is compiled with specific #defines. When this function breaks into the debugger, the debugger displays a "User breakpoint" message at exactly the spot of interest such that a detailed analysis can immediately take place. This technique has proven invaluable in tracking down extremely complex issues. One such difficult bug to catch is one that happens only under rare timing conditions in a multi-threaded application. This breakpoint function debugging capability has allowed FairCom engineers to bring more stability than ever to c-treeACE Server technology under these conditions.

Of course, there are some very simple steps you should take to help trap the majority of code bugs. These include:

• Compile with the -g or other applicable debug option to enable debugging symbols;
• Enabling Core/Dump Files (corelimit on Linux, minidump on Windows, crash analyzer on Mac);
• Attach the debugger directly to a running process and capture a stack trace or force a core dump;
• Set Breakpoints and watches;
• Enable ALL Warnings (-Wall)

Unix Based Tools

gdb and dbx are common debugging tools on Unix environments. Learning the many options these tools offer can really allow you to take full advantage of many advanced debugging techniques.

• set args and run <args> to run your executable with different command line options;
• gdb info threads. Great for picking through deadlocks;
• Step backwards through your program! (TRICKY!)

  Step backward with gdb

• Pretty printing: set print pretty on | off. Displays structures in human readable form.

Windows Visual Studio

Windows Visual Studio provides a full featured development environment with an advanced debugger built directly into the IDE. Enabling the right combination of call stack, watch and memory windows can allow you to watch data in your program processed in real time. Common windows you'll find on our developer screens:

• call stack window
• watch window
• memory window (watch your data directly in memory)
• threads window

A few other useful windows tips:

• Quickly identify Windows system errors (Tools->Error Lookup, or net helpmsg <error no> from a command prompt)
• Naming Threads

  MSDN -- Naming Threads

Visual Studio Tracing

Another productive technique is to set a breakpoint in strategic locations. Right click a breakpoint and choose "When Hit".

OR

Right click in the code window:

Define what action to take when the breakpoint is hit.

Collect Linux Back Traces from Hung Applications

Here's a cool tip for Linux if you're completely hung and have set your environment in advance.

Linux Debugging Tip

If your keyboard is remains functional and you have a hang on Linux, use the following method to help resolve the source of the hang problem. With these steps, you can display a back trace of the current running process and all processes using the magic key sequence.

1. The kernel that you are running must be built with CONFIG_MAGIC_SYS-REQ enabled. You must also be in text-mode. CTRL+ALT+F1 will put you into text mode, and CTRL+ALT+F7 will put you back in an X Windows session.
2. While in text-mode, press <ALT+ScrollLock> followed by <Ctrl+ScrollLock>. These magic keystrokes will give a stack trace of the currently running processes and all processes, respectively.
3. Look in your /var/log/messages file. If everything is correctly set, the system should have converted the symbolic kernel addresses for you. The back trace will be written to the /var/log/messages file.