There are many pages describing how to resample audio using the ffmpeg command line application, but what about doing resampling in your own program? To do that, you need to use the avcodec library (libavcodec.so on Linux and avcodec.dll on Windows).

1. Include avcodec.h
2. Call avcodec_init() to initialize the FFMpeg library.
3. Create a resampling context using av_resample_init() that describes how you want the resampling done.
4. Call av_resample() to do the actual resampling on your audio buffer.
5. When you’re done with the resampling context, delete it with av_resample_close().
6. Finally, link your application against avcodec, avutil, and zlib (it won’t work on Linux without this one).

Here it is in pseudocode:

#include "libavcodec/avcodec.h"

avcodec_init();

struct AVResampleContext* ctx = av_resample_init( ... );

av_resample( ctx, ... );

av_resample_close( ctx );

That’s it… seriously!

Sample Code (Linux):

Here’s a sample program I wrote that takes a raw 44.1kHz/16bit/mono audio file and plays it back using the pulseaudio API. The catch is that it allows you to specify a “skew” parameter which will cause the audio to dynamically speed up and slow down (via resampling). The amount of resampling is controlled by a sine wave, which is what drives the speed changes.

Download: resample.tar.bz

To unpack and build, type:

$ tar -xjvf resample.tar.bz
$ make

First, run the sample with no skew:

$ ./resample audio_16b_44k_mono_pcm_raw 0

Now, try it with a heavy skew:

$ ./resample audio_16b_44k_mono_pcm_raw -10000

For the thread “How VSync works, and why people loathe it“:

There is a technique called triple-buffering that solves this VSync problem. Lets go back to our 50FPS, 75Hz example. Frame 1 is in the frame buffer, and 2/3 of frame 2 are drawn in the back buffer. The refresh happens and frame 1 is grabbed for the first time. The last third of frame 2 are drawn in the back buffer, and the first third of frame 3 is drawn in the second back buffer (hence the term triple-buffering). The refresh happens, frame 1 is grabbed for the second time, and frame 2 is copied into the frame buffer and the first part of frame 3 into the back buffer. The last 2/3 of frame 3 are drawn in the back buffer, the refresh happens, frame 2 is grabbed for the first time, and frame 3 is copied to the frame buffer. The process starts over. This time we still got 2 frames, but in only 3 refresh cycles. That’s 2/3 of the refresh rate, which is 50FPS, exactly what we would have gotten without it. Triple-buffering essentially gives the video card someplace to keep doing work while it waits to transfer the back buffer to the frame buffer, so it doesn’t have to waste time. Unfortunately, triple-buffering isn’t available in every game, and in fact it isn’t too common. It also can cost a little performance to utilize, as it requires extra VRAM for the buffers, and time spent copying all of them around. However, triplebuffered VSync really is the key to the best experience as you eliminate tearing without the downsides of normal VSync (unless you consider the fact that your FPS is capped a downside… which is silly because you can’t see an FPS higher than your refresh anyway).

If the thread is ever unavailable, you can download a PDF version HERE.

ThreadNinja is a Linux library my team created that tracks pthread_create() and pthread_join() calls in an application. It prints a stacktrace where each thread is created and where it is joined. Any rogue (unjoined) threads are reported when the application exits. ThreadNinja is unobtrusive: it does NOT have to be compiled into the code. This means you can use it on applications you didn’t compile.

We found it useful and thought we’d share it. It’s be no means production code… just a tool. Hack on it, expand it, change it… whatever. It’s pretty small, so it should be easy to dive right in. We’ve released it under the BSD license.

Cut To The Chase

You can checkout the source code from Google Code, or download the version 1.0 tarball directly (threadninja.tar.gz).

To build ThreadNinja, simply untar it and call make:
> tar -zxf threadninja.tar.gz
> make

Now, simply use LD_PRELOAD to run the application:

> LD_PRELOAD=/path/to/threadninja/build/libthreadninja.so.1 TheApplication

If you don’t see function names in the stacktraces that are generated, then the application needs to be compiled with debug symbols. For my test app, I had to compile with the -rdynamic option:

> g++ -Wall -rdynamic main.cpp -lpthread

This causes the global symbol table to be included in the executable, which contains all the application’s function names. For more info, look at the --export-dynamic option on the GNU linker (ld) man page.

The Story Behind ThreadNinja

My team was assigned to stabilize a large video application that runs as a Linux-based appliance. The application consisted of 100,000+ of lines to code that was a tangle of build warnings, circular references, and many creative hacks. Our particular task was to fix a persistent set of seg-faults and memory leaks.

One of the first things we noticed was that the application didn’t shutdown properly. The shutdown logic was something akin to: signal components to exit, sleep 2 seconds, call Release() a couple times to clean up extra ref-counts, sleep 2 seconds, and then just call exit() to get the process to terminate (bypassing the remaining clean up code). Of course, this renders Valgrind useless when trying to find memory leaks, because the automatic memory cleanup code gets bypassed during the process abort.

As a result, our first priority was to get the app to shutdown cleanly. The first issue we ran into was that pthread_join() blocked indefinitely because threads were failing to terminate. We tried using GDB to track the threads, but many hundreds of threads were being created and destroyed dynamically. We needed a way to let the application run for hours, allow 1000+ threads to live and die, and still be able to track rogue threads. Hence, ThreadNinja was born.

How It Works

Through the magic of LD_PRELOAD, ThreadNinja “injects” itself between all calls to pthread_create() and pthread_join(). This happens because LD_PRELOAD instructs the loader to load ThreadNinja into memory first, before other libraries (in this case, before the pthread library). The result is that ThreadNinja’s implementation of pthread_create() and pthread_join() are used by the application instead of pthread’s own implementation. What ThreadNinja does is track the calls to these methods and then pass the call on to the “real” pthread implementation. From the application’s point-of-view, the behavior of the thread methods are the same… the tracking is transparent.

Output

Each time pthread_create() is called, a stacktrace and timestamp are printed to stdout:

Thread Created: 3047947120
[bt] Thu Jul 8 16:31:23 2010

[bt] ./a.out(StartService(unsigned long*, int))
[bt] ./a.out(Initialize())
[bt] ./a.out(main+0xb) [0x80489fa]
[bt] /lib/libc.so.6(__libc_start_main+0xe6) [0xb5bbb6]
[bt] ./a.out() [0x8048811]

The first line (“Thread Created”) gives the value of the pthread_t handle, so you can later track where rogue threads where created. The next line is the time when the create happened. The remaining lines are the call stack that led to the pthread_create() call.

Each time pthread_join() is called, similar information is printed to stdout:

Thread Joined: 3047947120
[bt] Thu Jul 8 16:31:31 2010

[bt] ./a.out(Terminate())
[bt] ./a.out(main+0x10) [0x80489ff]
[bt] /lib/libc.so.6(__libc_start_main+0xe6) [0xb5bbb6]
[bt] ./a.out() [0x8048811]

When the application terminates (cleanly or uncleanly), a summary of the current state of the application threads is printed:

exit_handler()
[Thread Summary]
Total Created: 573
Total Joined: 568
Total Running: 5

In this case, you’ll notice that 5 threads were never joined on.

Limitations

ThreadNinja only tracks calls to pthread_create() and pthread_join(). This means calls like system(), exec(), and fork() are not tracked. Also, calls to pthread_cancel() are not tracked. We had started adding code to track pthread mutexes and stuff, but it turned out we didn’t need it. Feel free to add support for all this stuff and submit changes to the Google code site.

Happy coding!

Wrong.

I think Bruce Schneier described it best (paraphrased): Cryptography is like having a really strong front door on your house… 2 foot thick steal, blast proof, the whole 9 yards. A thief isn’t going to try and break through your front door… they’ll just climb through a window!

Security is about the whole system; not just the crypto. xkcd summed it up nicely:

Note: I’m talking here about interval timing (i.e. accurately measuring the duration between 2 events). This is different than synchronizing different clocks or maintaining accurate wall time.

Anatomy of the Clock

The first mistake most people make when doing timing is to use functions like gettimeofday(), GetSystemTime(), etc. These functions return what is called “wall time”… time that corresponds to a calendar date/time. These clocks suffer from the follow limitations:

1. They have a low resolution: “High-performance” timing, by my definition, requires clock resolutions into the microseconds or better.
2. They can jump forwards and backwards in time: Computer clocks all tick at slightly different rates, which causes the time to drift. Most systems have NTP enabled which periodically adjusts the system clock to keep them in sync with “actual” time. The adjustment can cause the clock to suddenly jump forward (artificially inflating your timing numbers) or jump backwards (causing your timing calculations to go negative or hugely positive).

For interval timing, all that’s needed for a clock is a simple counter that increments at a stable rate. For high-performance timing, the rate this counter increments should be high. A related constraint is that the counter must be monotonic (can never “tick” backwards… ever). The counter may overflow and wrap back to 0, but using unsigned math in your timing calculations can compensate for that (see example below).

Something to note: since we are usually measuring short durations, the drift of the clock is so small that we aren’t concerned by it (what matters is the drift between successive reads, not total drift over time).

Using the Clock

To use the clock, you need access to 2 pieces of information: the clock value, and the rate (frequency) at which it increments. Once you have that information, it’s trivial to calculate the duration between 2 successive reads of the clock.

For example, consider the following:

-        start equals the clock value at the beginning of the interval to measure.

-        end equals the clock value at the end of the interval.

-        frequency equals the frequency that the clock increments per second.

Calculating the duration of the interval (in seconds) is as simple as:

duration = (end – start) / frequency

duration will equal the floating-point interval time in seconds (e.g. 0.000237 = 237us).

A common use case I’ve seen is wanting to measure the duration between successive calls to a function. Here’s one way to implement that:

float freq = (float) get_frequency();

Foo()
{
unsigned now = read_clock();
float duration = (float)(now – last) / (float)freq;
last = now;
}

It is important to use unsigned variables for now and last or your calculations will go haywire if the clock ever wraps back to 0. Consider what happens if now is less than last. Using unsigned variables allows the math operation to underflow, which produces the correct result. If you don’t believe me, just write some test code and see for yourself… it’s an important concept to understand.

Don’t Use RDTSC As Your Clock

I’m dismayed by how many forum posts suggest that newbie’s use RDTSC for timing. Don’t get me wrong; the TSC isn’t bad in-and-of-itself. It’s just way too hard to get timing right with it unless you’re an expert… and if you’re reading this, chances are you aren’t . Stick to the higher level API’s I present below.

Even if you are an expert, I still suggest avoiding RDTSC for the following reasons:

1. It’s processor core specific: Using RDTSC will give different values on different processor cores. This causes non-monotonic clock behavior as a thread is migrated across cores during execution (i.e. the clock can tick backwards on two successive reads). Conversely, the clock may also appear to jump forwards if the thread moves to a different core.
2. It doesn’t always tick at the same rate: On some systems, the clock frequency will vary as the CPU load changes. This is due to power saving features in the processor that throttle down the clock speed when the load is low (e.g. Intel SpeedStep).
3. It’s unnecessary: there are better alternatives to RDTSC that are readily available (HPET and APIC).

“Game Timing and Multicore Processors” has more info, though it’s centered around Windows.

“Guidelines For Providing Multimedia Support” has a good summary of the hardware clocks on the PC platform, and justification for the creation of the HPET.

A more technical look into the past problems with RDTSC can be found in this email from Rick Brunner (AMD Fellow).

If you’re having trouble sleeping, you can look at Intel’s HPET design document.

Linux Clock

POSIX.1b defines realtime clock methods that you’ll find on most *NIX systems (the full spec can be viewed HERE). Specifically, you want to use clock_getres() and clock_gettime(). clock_getres() returns the resolution (frequency) of the clock, and clock_gettime() returns the current value of the clock. Most systems implement the CLOCK_MONOTONIC type, which provides a frequency-stable, monotonically-increasing counter. The resolution of CLOCK_MONOTONIC is high on the 2.6 kernel, in my experience. I recommend using this clock when building a high-performance timing solutions on Linux.

The methods are defined in time.h, and you need to link against librt (pass ‘-lrt’ to gcc). The prototypes for the functions are:

int clock_getres(clockid_t clock_id, struct timespec *res);
int clock_gettime(clockid_t clock_id, struct timespec *tp);

A detailed description of these methods can be found HERE.

I also suggest looking at clock_nanosleep(), but that’s a separate topic.

Windows Clock

On Windows, QueryPerformanceFrequency() and QueryPerformanceCounter() are the obvious choice. QueryPerformanceFrequency() returns (surprise!) the frequency of the counter. QueryPerformanceCounter() returns the current value of the counter. Just like CLOCK_MONOTONIC on Linux, the Windows performance counter is a high-frequency, stable, monotonically-increasing counter.

The methods are defined in windows.h. The prototypes are:

BOOL QueryPerformanceFrequency(LARGE_INTEGER *lpFrequency);
BOOL QueryPerformanceCounter(LARGE_INTEGER *lpPerformanceCount);

Windows Timing Errata

Note: The follow items do NOT affect the resolution of the Window’s performance counter, but I think it’s still important to know when doing timing on Windows.

One problem you may run into on Windows is that the default system clock interval defaults to 10 or 15ms (depending of the OS version). This clock is what drives all the timers and sleep functions for that platform. What this means is that, left to the default, your timers will trigger 5 to 7.5ms late, on average.

You can fix this by increasing the system clock resolution to 2ms (I remember reading a tech article by Microsoft saying that 2ms gave better system performance than 1ms, but I can’t find it for the life of me). You do this by using the timeGetDevCaps(), timeBeginPeriod() and timeEndPeriod(). Sample code can be found HERE. You have to include mmsystem.h and link against Winmm.lib.

I feel it necessary to note that increasing the system clock interval negatively affects power consumption. The Windows 7 blog as an interesting breakdown of “Windows 7 Energy Efficiency”. Specifically, they noticed a 10% drop in battery life when the clock resolution was set to 1ms using timeBeginPeriod().

Maybe this last section belongs in a separate post, but whatever…

Example

Consider the following log method:

void WriteLog( const LOG_LEVEL level, const char* file, const int line, const char* format, ... );

Pretty straight-forward: it allows you to specify the log level (NOTICE, WARNING, ERROR, etc), a file name and line number, and a user-defined description in “printf” format. To log a warning message, you would type:

WriteLog( LL_WARNING, __FILE__, __LINE__, "[WARNING] Failed to import descriptor '%s:%i'!", _descriptor, _id );

The entry in the log file would look like:

Core.cpp:715 [WARNING] Failed to import descriptor 'timing-engine'!

Creating a Variadic Macro

Calling WriteLog directly works fine, but it’s pretty verbose and annoying to use. One way to make it simpler is to wrap WriteLog with a variadic macro:

#define LOG_WARNING( format, ... ) \\
WriteLog( LL_WARNING, __FILE__, __LINE__, "[WARNING] " format, ##__VA_ARGS__ );

Then the example above would become:

LOG_WARNING( "Failed to import descriptor '%s:%i'!", _descriptor, _id );

The macro automatically fills in the file and line number where the log message was generated, and the log level is specified by the macro name (LOG_WARNING). This is much more straight-forward and I believe it makes the code easier to read.

Notice that __VA_ARGS__ is used to get the variable argument list and pass it to WriteLog. During compilation, the preprocessor replaces __VA_ARGS__ with the comma-separated list of  arguments. The ‘##‘ that prefixes __VA_ARGS__ is vital if you want your macro to work like you expect. Without it, you would be required to have at least one argument after the format string. ’##‘ has a special meaning in this case: it causes the preceding comma to be deleted if there are no variable arguments. Without this, the following line would generate a syntax error:

LOG_WARNING( "This would cause a syntax error" );

Prefixing __VA_ARGS__ with ‘##‘ allows the above code to work just fine.

Looking at the logs, we found that our audio pipeline was failing to open /dev/dsp on the restart. We then used lsof to list the open file descriptors to see which process currently held /dev/dsp:

# lsof | grep /dev/dsp
ntpd   18857    root   16u    CHR     14,3    180099 /dev/dsp


What!?!?… why the heck is NTP opening the sound device and how did it steal it from us??? After some discussion we started remembering a problem in the past with ntpd stealing our SNMP diagnostics port. This just didn’t make any sense.

Digging into our appliance code, we found this line:

system( "service ntpd restart" );

This would be called each time we were notified by the security system that the NTP server address had changed (which fired once each time the process was started so we could get the initial address). But this still didn’t explain why NTP took over ownership of our file descriptors on restart.

Long story short: system() is implemented as fork() followed by execv(). By default, fork() gives a copy of the parent’s file descriptors to the child process (i.e. the ntpd child process got a copy of the /dev/dsp file descriptor). To prevent this, you have to set the FD_CLOEXEC flag on the file desciptors you don’t want copied.

For example:

fd = open( "/dev/dsp", O_RDWR );
fcntl( fd, F_SETFD, FD_CLOEXEC );

Conclusion: setting the FD_CLOEXEC flag on the /dev/dsp file descriptor fixed the problem for audio. However, most of the other file desciptors still got owned by ntpd. Did we go back and set the FD_CLOEXEC flag on all file descriptors, you ask? Nope. It turns out we had a script monitoring the NTP config file and restarting ntpd for us when the file got updated… we just had to update the config file and remove the system( "service ntpd restart" ) call.

Oh, and the reason audio worked on first boot but not subsequent restarts was due to a weird race condition around when /dev/dsp got opened.

It displayed remarkable facility with cultural trivia (“This action flick starring Roy Scheider in a high-tech police helicopter was also briefly a TV series” — “What is ‘Blue Thunder’?”), science (“The greyhound originated more than 5,000 years ago in this African country, where it was used to hunt gazelles” — “What is Egypt?”) and sophisticated wordplay (“Classic candy bar that’s a female Supreme Court justice” — “What is Baby Ruth Ginsburg?”).

Software firms and university scientists have produced question-answering systems for years, but these have mostly been limited to simply phrased questions. Nobody ever tackled “Jeopardy!” because experts assumed that even for the latest artificial intelligence, the game was simply too hard: the clues are too puzzling and allusive, and the breadth of trivia is too wide.

With Watson, I.B.M. claims it has cracked the problem — and aims to prove as much on national TV. The producers of “Jeopardy!” have agreed to pit Watson against some of the game’s best former players as early as this fall (emphasis mine). To test Watson’s capabilities against actual humans, I.B.M.’s scientists began holding live matches last winter.

I’d definitely watch that episode… especially if Watson was pitted against Ken Jennings.

Under the hood:

[IBM's] main breakthrough was not the design of any single, brilliant new technique for analyzing language. Indeed, many of the statistical techniques Watson employs were already well known by computer scientists. One important thing that makes Watson so different is its enormous speed and memory. Taking advantage of I.B.M.’s supercomputing heft, Ferrucci’s team input millions of documents into Watson to build up its knowledge base — including, he says, “books, reference material, any sort of dictionary, thesauri, folksonomies, taxonomies, encyclopedias, any kind of reference material you can imagine getting your hands on or licensing. Novels, bibles, plays.”

The full article is worth the read.