ThreadNinja is a Linux library my team created that tracks pthread_create() and pthread_join() calls in an application. It prints a stacktrace where each thread is created and where it is joined. Any rogue (unjoined) threads are reported when the application exits. ThreadNinja is unobtrusive: it does NOT have to be compiled into the code. This means you can use it on applications you didn’t compile.

We found it useful and thought we’d share it. It’s be no means production code… just a tool. Hack on it, expand it, change it… whatever. It’s pretty small, so it should be easy to dive right in. We’ve released it under the BSD license.

Cut To The Chase

You can checkout the source code from Google Code, or download the version 1.0 tarball directly (threadninja.tar.gz).

To build ThreadNinja, simply untar it and call make:
> tar -zxf threadninja.tar.gz
> make

Now, simply use LD_PRELOAD to run the application:

> LD_PRELOAD=/path/to/threadninja/build/libthreadninja.so.1 TheApplication

If you don’t see function names in the stacktraces that are generated, then the application needs to be compiled with debug symbols. For my test app, I had to compile with the -rdynamic option:

> g++ -Wall -rdynamic main.cpp -lpthread

This causes the global symbol table to be included in the executable, which contains all the application’s function names. For more info, look at the --export-dynamic option on the GNU linker (ld) man page.

The Story Behind ThreadNinja

My team was assigned to stabilize a large video application that runs as a Linux-based appliance. The application consisted of 100,000+ of lines to code that was a tangle of build warnings, circular references, and many creative hacks. Our particular task was to fix a persistent set of seg-faults and memory leaks.

One of the first things we noticed was that the application didn’t shutdown properly. The shutdown logic was something akin to: signal components to exit, sleep 2 seconds, call Release() a couple times to clean up extra ref-counts, sleep 2 seconds, and then just call exit() to get the process to terminate (bypassing the remaining clean up code). Of course, this renders Valgrind useless when trying to find memory leaks, because the automatic memory cleanup code gets bypassed during the process abort.

As a result, our first priority was to get the app to shutdown cleanly. The first issue we ran into was that pthread_join() blocked indefinitely because threads were failing to terminate. We tried using GDB to track the threads, but many hundreds of threads were being created and destroyed dynamically. We needed a way to let the application run for hours, allow 1000+ threads to live and die, and still be able to track rogue threads. Hence, ThreadNinja was born.

How It Works

Through the magic of LD_PRELOAD, ThreadNinja “injects” itself between all calls to pthread_create() and pthread_join(). This happens because LD_PRELOAD instructs the loader to load ThreadNinja into memory first, before other libraries (in this case, before the pthread library). The result is that ThreadNinja’s implementation of pthread_create() and pthread_join() are used by the application instead of pthread’s own implementation. What ThreadNinja does is track the calls to these methods and then pass the call on to the “real” pthread implementation. From the application’s point-of-view, the behavior of the thread methods are the same… the tracking is transparent.

Output

Each time pthread_create() is called, a stacktrace and timestamp are printed to stdout:

Thread Created: 3047947120
[bt] Thu Jul 8 16:31:23 2010

[bt] ./a.out(StartService(unsigned long*, int))
[bt] ./a.out(Initialize())
[bt] ./a.out(main+0xb) [0x80489fa]
[bt] /lib/libc.so.6(__libc_start_main+0xe6) [0xb5bbb6]
[bt] ./a.out() [0x8048811]

The first line (“Thread Created”) gives the value of the pthread_t handle, so you can later track where rogue threads where created. The next line is the time when the create happened. The remaining lines are the call stack that led to the pthread_create() call.

Each time pthread_join() is called, similar information is printed to stdout:

Thread Joined: 3047947120
[bt] Thu Jul 8 16:31:31 2010

[bt] ./a.out(Terminate())
[bt] ./a.out(main+0x10) [0x80489ff]
[bt] /lib/libc.so.6(__libc_start_main+0xe6) [0xb5bbb6]
[bt] ./a.out() [0x8048811]

When the application terminates (cleanly or uncleanly), a summary of the current state of the application threads is printed:

exit_handler()
[Thread Summary]
Total Created: 573
Total Joined: 568
Total Running: 5

In this case, you’ll notice that 5 threads were never joined on.

Limitations

ThreadNinja only tracks calls to pthread_create() and pthread_join(). This means calls like system(), exec(), and fork() are not tracked. Also, calls to pthread_cancel() are not tracked. We had started adding code to track pthread mutexes and stuff, but it turned out we didn’t need it. Feel free to add support for all this stuff and submit changes to the Google code site.

Happy coding!

Looking at the logs, we found that our audio pipeline was failing to open /dev/dsp on the restart. We then used lsof to list the open file descriptors to see which process currently held /dev/dsp:

# lsof | grep /dev/dsp
ntpd   18857    root   16u    CHR     14,3    180099 /dev/dsp


What!?!?… why the heck is NTP opening the sound device and how did it steal it from us??? After some discussion we started remembering a problem in the past with ntpd stealing our SNMP diagnostics port. This just didn’t make any sense.

Digging into our appliance code, we found this line:

system( "service ntpd restart" );

This would be called each time we were notified by the security system that the NTP server address had changed (which fired once each time the process was started so we could get the initial address). But this still didn’t explain why NTP took over ownership of our file descriptors on restart.

Long story short: system() is implemented as fork() followed by execv(). By default, fork() gives a copy of the parent’s file descriptors to the child process (i.e. the ntpd child process got a copy of the /dev/dsp file descriptor). To prevent this, you have to set the FD_CLOEXEC flag on the file desciptors you don’t want copied.

For example:

fd = open( "/dev/dsp", O_RDWR );
fcntl( fd, F_SETFD, FD_CLOEXEC );

Conclusion: setting the FD_CLOEXEC flag on the /dev/dsp file descriptor fixed the problem for audio. However, most of the other file desciptors still got owned by ntpd. Did we go back and set the FD_CLOEXEC flag on all file descriptors, you ask? Nope. It turns out we had a script monitoring the NTP config file and restarting ntpd for us when the file got updated… we just had to update the config file and remove the system( "service ntpd restart" ) call.

Oh, and the reason audio worked on first boot but not subsequent restarts was due to a weird race condition around when /dev/dsp got opened.

Fault bucket 42424242, type 1
Event Name: APPCRASH
Response: None
Cab Id: 0

Problem signature:
P1: MyApp.exe
P2: 1.42.42.42
P3: 598773cf
P4: StackHash_ac62
P5: 0.0.0.0
P6: 00000000
P7: c0000007
P8: 00000000
P9:
P10:

We spent some time wondering if our crypto libraries were the problem (we just made some changes recently), but concluded that was unlikely. So what the heck is the “StackHash” module? Did our trashed stack cause the kernel to think we were a different module? Nope.

The answer is that the Windows executive couldn’t identify the module we were in when the application crashed (it uses the instruction pointer to determine what code was executing). In this case, the kernel simply takes a hash of the stack so at least we might be able to identify if we’ve seen this exact crash before. Here’s the answer summarized by an engineer from Microsoft:

In the OS when I try to get a faulting module name it is possible that there is no module laoded (sic) at that address. For example in this case the EIP was zero. So in those cases where a module is not loaded and it is not also in the unloaded module list, I take a stack hash of the stack so that we can identify this crash from other crashes where also the module is not known.

First, you need to include execinfo.h to your code:

#include <execinfo.h>

Next, call the backtrace() function to get an array of void pointers that represents the current stack (the pointers are the return addresses for each stack frame).

void* tracePtrs[100];
int count = backtrace( tracePtrs, 100 );

The backtrace() function returns the number of entries in the array (read the man pages for more info about the array size).

Finally, you need to resolve the function names associated with the pointers. You have 2 options: backtrace_symbols() and backtrace_symbols_fd(). Both of these methods resolve the pointers to strings, but the difference is that backtrace_symbols() allocates the strings on the heap while backtrace_symbols_fd() writes the strings to a file descriptor that you can read. Just keep in mind that backtrace_symbols() won’t work if the heap has been trashed.

Here’s an example using backtrace_symbols():

char** funcNames = backtrace_symbols( tracePtrs, count );

// Print the stack trace
for( int ii = 0; ii < count; ii++ )
    printf( "%s\n", funcNames[ii] );

// Free the string pointers
free( funcNames );

NOTE: Make sure you call free() on the array of strings returned from backtrace_symbols().

For more information, here’s a good article from the Linux Journal.

#define _CRTDBG_MAP_ALLOC
// Include other header files here
#include <crtdbg.h>

Then, you call _CrtDumpMemoryLeaks() before your application exits. If your program exits at many points, you can alternatively call _CrtSetDbgFlag( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ) at the beginning of you application, which will cause the leaks to also be printed when it exits. The results are printed to the Debug Window and look like the following:

Detected memory leaks!
Dumping objects ->
C:\PROGRAM FILES\VISUAL STUDIO\MyProjects\leaktest\leaktest.cpp(20) :
{18} normal block at 0x00780E80, 64 bytes long.
Data: <                > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD
Object dump complete.

Cool, Huh?! However, some libraries don’t play nice with this, as I explain below.

Memory Leak Detection and ACE:

I’ve mentioned ACE before in a previous post evaluating portable runtime environments. It’s a pretty cool set of libraries that provides a portable OS abstraction layer and communication environment. If you write your application using ACE, it will compile and run on a whole host of system types… but I digress.

One problem with ACE is that it conflicts with the Windows leak check code. If you define _CRTDBG_MAP_ALLOC before the ACE headers, you will get build errors like the following:

1>..\include\ace\os_ns_unistd.h(154) : error C2059: syntax error : 'constant'

What’s happening is that the ACE headers try to enable the leak detection code themselves, which causes the overridden memory functions to conflict with the original versions. In essence, ACE is trying to define malloc (for example) after crtdbg.h has already overridden it.

So, how do you get your application to build with leak detection and ACE? Define _CRTDBG_MAP_ALLOC after the ACE headers (Remember: the ACE headers must always be included before other headers… another complaint I have).

#include "ace/ACE.h"
#define _CRTDBG_MAP_ALLOC
// Include other headers here
#include <crtdbg.h>

Okay, your application should compile now. Run it and look at the output window. For a simple application that does nothing but return, this is what I see:

Detected memory leaks!
Dumping objects ->
{178} normal block at 0x00CDBD78, 50 bytes long.
Data: <---TEST---      > 2D 2D 2D 48 45 4C 4C 4F 2D 2D 2D 00 CD CD CD CD
{173} normal block at 0x002EAAC8, 8 bytes long.
Data: <  .     > C0 A1 2E 00 CD CD CD CD
{172} normal block at 0x002EAA78, 20 bytes long.
Data: <4  Z       Z    > 34 90 F3 5A 0A 01 00 00 F3 17 DA 5A 03 00 00 00
{171} normal block at 0x002EAA18, 32 bytes long.
Data: <   Z  ]         > C4 91 F3 5A C0 FB 5D 00 FF FF FF FF 00 00 00 00
{170} normal block at 0x002EA9B8, 32 bytes long.
Data: <   Z  ]         > C4 91 F3 5A 88 FB 5D 00 FF FF FF FF 00 00 00 00
{169} normal block at 0x002EA958, 32 bytes long.
Data: <   ZP ]         > B4 91 F3 5A 50 FB 5D 00 FF FF FF FF 00 00 00 00
{168} normal block at 0x002EA910, 8 bytes long.
Data: <   Z    > D4 91 F3 5A 00 00 00 00

... Truncated for sanity (30 more leaks follow) ...

Object dump complete.
The program '[14288] simple.exe: Native' has exited with code 0 (0x0).

These are very likely false-positives, but they are annoying. There are other problems too:

1. There are no file/line numbers printed, so we can’t track down the code to verify if the errors are real or not.
2. File and line numbers don’t show up for errors outside of ACE (the first leak is mine).
3. All this output adds noise that can hide other memory leaks (I wish Microsoft would add a way to suppress errors we don’t care about… props to Valgrind for doing this).

Anyways, all of this is quite frustrating. I was hunting around on the ACE newsgroup for solutions and could only find this thread:

“In general, we track memory issues on Windows using Purify, so you might try going that route.”
Douglas Schmidt

“Correct. This memory tracking scheme and the ACE restrictions onheader file placement are at odds here. You have two choices:

1. Use a different memory tracking scheme such as Purify, which Doug
suggested.

2. Develop, or sponsor, necessary changes to remove ACE’s restrictions
on header file placement.”
Steve Huston

So there you have it: ACE and Window memory leak detection don’t play nice together. I hate encountering a problem without a viable solution… it bugs me (no pun intended). I even tried modifying the ACE headers with marginal results (it came close to compiling). Oh well…

Breakpoint Basics:

Setting a breakpoint in GDB is supposed to be simple. Here we set a breakpoint at line 50 in file main.cpp:

(gdb) b main.cpp:50
Breakpoint 1 at 0x804937a: file main.cpp, line 50.

We can also use the function name and GDB will attempt to find the correct location for us:

(gdb) b DoSomething
Breakpoint 2 at 0×8049334: file main.cpp, line 150

Simple, right? Just wait…

Breakpoint Gotchas:

GDB’s breakpoint logic is pretty handy for simple projects, but it can break down fast when things get more complicated.

For example, let’s say your application is plugin-driven, with each plugin being a separate library. Now assume each plugin has a Plugin.cpp file under it’s own Source directory. Try to set a breakpoint in the Initialize() method of the Plugin class:

(gdb) b Initialize
Breakpoint 3 at 0×8049717: file main.cpp, line 230

Oops! There is an Initialize() method in main.cpp and GDB thought that’s where we wanted to put it: wrong!

Okay, let’s be more specific:

(gbd) b Plugin::Initialize
Breakpoint 4 at 0×8046194: file Source/Plugin.cpp, line 89

Okay, that looks better, but which Plugin.cpp file did GDB put the breakpoint? How do we know it’s the file for the plugin we want?

If we used namespaces, then we can get more specific:

(gdb) b MY_PLUGIN_A::Plugin::Initialize
Breakpoint 5 at 0×8050039: file Source/Plugin.cpp, line 130

We can see from the address and line number that the previous breakpoint was at the wrong place. Okay, moving on…

Setting Breakpoint in Templates:

Templates can be much harder to set breakpoints in because we have to specify the exact prototype for the fully-defined template. We as programmers are used to the compiler handling the template type stuff for us, so it can be difficult to guess the correct type.

For example, assume we have some abstract class BarAbstract that uses the template Foo<> to make it concrete. Now assume we’re really clever and we want to hide this from the users of our class. We could use a typedef to hide the true type:

typedef Foo<BarAbstract> Bar;

Now, all the user needs to do is instantiate Bar without a thought to the Foo<> template.

int DoSomething()
{
Bar b;
return b.Baz();
}

So far so good? Okay, now how the heck do you set a breakpoint in the Baz() method of class Bar?

The quick answer: use objdump, c++filt, and grep to find the complete definition that GDB will need.

$ objdump -t libMyLib.so | c++filt | grep ‘BarAbstract.*Baz’
0000d2d6 w F .text 0000000a     MY_PLUGIN_A::Foo<MY_PLUGIN_A::BarAbstract>::Baz()

Now, simply copy-n-paste the full method definition to GDB when setting the breakpoint:

(gdb) b MY_PLUGIN_A::Foo<MY_PLUGIN_A::BarAbstract>::Baz()
Breakpoint 6 at 0×8048890: file Source/Bar.cpp, line 355

That’s it! Happy debugging!