We ran into a weird problem the other day where our Linux video display appliance would lose audio support when the process was restarted. The audio was supposed to play through a custom joystick-keyboard that was attached via USB (the keyboard is used by security guards to PTZ cameras, control monitors, etc). The audio could be heard just fine when the box first booted, but if the application restarted audio would be lost.
Looking at the logs, we found that our audio pipeline was failing to open /dev/dsp on the restart. We then used lsof to list the open file descriptors to see which process currently held /dev/dsp:
# lsof | grep /dev/dsp
ntpd 18857 root 16u CHR 14,3 180099 /dev/dsp
What!?!?… why the heck is NTP opening the sound device and how did it steal it from us??? After some discussion we started remembering a problem in the past with ntpd stealing our SNMP diagnostics port. This just didn’t make any sense.
Digging into our appliance code, we found this line:
system( "service ntpd restart" );
This would be called each time we were notified by the security system that the NTP server address had changed (which fired once each time the process was started so we could get the initial address). But this still didn’t explain why NTP took over ownership of our file descriptors on restart.
Long story short: system() is implemented as fork() followed by execv(). By default, fork() gives a copy of the parent’s file descriptors to the child process (i.e. the ntpd child process got a copy of the /dev/dsp file descriptor). To prevent this, you have to set the FD_CLOEXEC flag on the file desciptors you don’t want copied.
For example:
fd = open( "/dev/dsp", O_RDWR );
fcntl( fd, F_SETFD, FD_CLOEXEC );
Conclusion: setting the FD_CLOEXEC flag on the /dev/dsp file descriptor fixed the problem for audio. However, most of the other file desciptors still got owned by ntpd. Did we go back and set the FD_CLOEXEC flag on all file descriptors, you ask? Nope. It turns out we had a script monitoring the NTP config file and restarting ntpd for us when the file got updated… we just had to update the config file and remove the system( "service ntpd restart" ) call.
Oh, and the reason audio worked on first boot but not subsequent restarts was due to a weird race condition around when /dev/dsp got opened.
