Note: This is intended to be a pragmatic guide for engineers evaluating codecs. It is not a comprehensive treatment of the subject. The goal is to give readers a solid overview and some practical ideas.

Get Familiar with Psychoacoustics

Psychoacoustics is the study of how humans perceive sound. As you might expect, we humans don’t process sound in a perfect, linear fashion. The physical shape of the ear, the transfer function of the Basilar membrane, and the psychological interpretation of the data all affect how we perceive sound (and by extension, how “good” an audio codec sounds to us).

I highly recommend you start by reading this excerpt from Surround Sound: Psychoacoustics Part 1, by Tomlinson Holman (he created THX for Lucasfilm).

Understand the Codec

Make sure you understand the codec you are testing; not necessarily the implementation, but what tools (i.e. methods) the codec uses for compression. Many codecs have different “profiles”, which describe what subset of available tools are used (e.g. AAC). You should also have some idea how each compression tool works and any short-comings it has. This will help guide you in selecting reference audio samples and knowing what artifacts to listen for.

For an introduction to modern audio compression, read Audio Coding: An Introduction to Data Compression Part 1, and Part 2 (discusses MP3 and AAC). I actually suggest buying the book “Introduction to Data Compression”, by Khalid Sayood.

Understand the API

Make sure you actually read the codec documentation and look at any available code samples. This step has more to do with due-diligence than anything, as I haven’t seen a codec API we couldn’t work with, but you need to do this. This will also help you scope the work required to get a working encoder/decoder for future steps (if your lucky, the sample code can be used).

Choose the Reference Audio Samples

An effective test requires multiple audio samples with different characteristics. There are many types of artifacts a codec can introduce, and your choice of audio samples will dictate how easy they are to detect. It’s also important to pick samples that reflect the actual types of sound the codec to have to deal with. For example, if the final system will primarily be encoding speech, then you should choose more speech-oriented references as opposed to music samples.

Some characteristics you might consider:

• Transients (snare drum): Sensitive to pre-echo and noise “smearing”.
• Tonal structure (clarinet, saxophone): Sensitive to noise and “roughness”.
• Natural speech (male and female voices of various languages): Sensitive to distortion and smearing of “attacks”.
• Complex sound (bag pipes): Stresses the codec.
• High bandwidth (bag pipes): Loss of high frequencies and program-modulated high frequency noise.

It is also possible to use synthetic sounds and sweeps, but this is only recommended for the automated objective tests below.

As a basic guideline, you need 10-25 second “raw” samples recorded at the highest sample rate your system needs to work with. It is vital that the samples you choose have never been compressed with a lossy codec (mp3, AAC, etc)… that would severally limit the quality of your test. For sample rate and size, I suggest 48kHz 16-bit PCM, but a lower rate/size makes sense if the final system is limited in this area. It also makes sense to use a sample rate of 44.1kHz, since many quality audio samples can be ripped losslessly from CD. Just keep in mind that the objective PEAQ test mentioned below requires 48kHz 16-bit PCM, so up-sampling may be required.

The audio samples can be stored in whatever container format you want (raw, WAV, etc) as long as your codec test application can unpack it. This is important to keep in mind… you don’t want to accidentally run the WAV header through the codec (yes, I’ve done this). The container format is more of a practical issue, but it was worth mentioning.

Generate Various Test Samples and Observe CPU Load

This step is pretty straight-forward: wrap the codec in an application and encode the reference audio samples at different bit rates. You should choose bit rates that represent the full spectrum of bit rates that will be used in the final system. While you’re encoding, track the CPU usage on the codec and how many cores it’s using. You may even want to do a separate test running many encodes in parallel (this works nicely if the CPU usage is too low to measure accurately). Make sure to consider application overhead and disk I/O when making measurements.

After encoding, you need to decode back to raw PCM. Clearly label your files so you know what bit rate each one was encoded with. These decoded test samples are what we will be comparing to the original reference samples.

Do a Subjective Test

How you conduct your subjective testing will depend on several factors, such as time constraints, cost, and the required test precision. At the low end, you could simply listen to the test samples in a pair of headphones and judge the quality yourself. For a high precision test, you could do a full ITU BS.1116 test using “expert listeners” in a controlled environment. While these examples represent the extremes, there are many permutations that can give you the desired quality of results.

The most common subjective test is called a “double-blind triple-stimulus with hidden reference” test. The listener hears three samples (commonly labeled A, B, and C) for a period of 10 to 25 seconds. A is always the original reference sample. The next two samples, B and C, are randomly assigned either the test sample from the codec or the original reference sample played again (called the “hidden reference”). The listener must then rate the difference between B and A, and C and A, not knowing which one is the test sample. The grading scale is:

• 5.0 Imperceptible
• 4.0 Perceptible, but not annoying
• 3.0 Slightly annoying
• 2.0 Annoying
• 1.0 Very annoying

Ideally, you would conduct several tests and average the results together. If you do the listening test yourself, your results will be limited to your listening skills and understanding of audio codec artifacts. Here’s a summary of factors that affect the quality of your results:

• The quality of the listener.
• The choice of audio samples.
• The number and duration of the testing.
• The testing environment, including speaker/headphone quality, room design, and listener placement.
• The quality of randomization of sample order to remove any correlation between samples.
• Proper statistical analysis of the combined test results.

A proper subjective test is both expensive and time consuming. It’s important to find the right balance for your particular needs.

Do an Objective Test

Evaluating a codec objectively requires testing methods that correlate well to actual human perception. You can’t simply measure the distortion introduced by the codec using traditional measurements like Signal-to-Noise ratio (S/N) and Total-Harmonic-Distortion (THD), because they don’t correlate well to perceived audio quality. Some distortion is imperceptible to the human ear, and codecs take advantage of this to increase the compression ratio.

Fortunately, the ITU has standardized an objective audio test called PEAQ (BS.1387). The acronym stands for Perceptual Evaluation of Audio Quality. PEAQ uses software to model the entire human auditory system (including blood flow noise in the inner ear) to generate a set of metrics that are used to give a final “quality” score. The original reference signal is compared to a signal run through the codec, and the result is a real number between 0.0 and -4.0. The result is interpreted on the following scale:

• 0.0 = Imperceptible
• -1.0 = Perceptible but not annoying
• -2.0 = Slightly annoying
• -3 .0= Annoying
• -4.0 = Very annoying

Obviously, values closer to zero are better.

The test was developed by a similar group of audio experts that developed BS.1116 (mentioned above) and the results have been validated against a long list of subjective tests done using expert listeners.

There are several free and commercial software packages available for doing PEAQ tests. The best free package I’ve found is AFsp from the McGill Telecommunications and Signal Processing Lab. There’s also peaqb, but there’s a comment that it gives incorrect results. AFsp worked great in my tests and included some helpful tools like CompAudio and InfoAudio.

Summary

Hopefully this post has given you a good starting point and some practical ideas for testing audio codecs. My goal was to provide a pragmatic approach with different options depending on what your actual evaluation needs are. This is in no way a comprehensive treatment of the subject; only an overview. I highly suggest reading some of the books I referenced if you’d like a deeper treatment of the subject. Either way, I hope you found this post helpful.

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

In summary:

• Everything you tell the police can be used AGAINST you, but it can NEVER be used to help you (because it’s hear-say at that point).
• There is no way talking to the police can help you.
• You may admit guilt (even if innocent) with no benefit in return.
• Even if you are innocent, it is easy to get carried away and tell a small lie, which can destroy your credibility.
• Even if you are innocent, and only tell the truth, you will always give the police information that can help convict you.
• Even if you are innocent, only tell the truth, and say nothing incriminating, the police may not recall the conversation with 100% accuracy.
• Even if you are innocent, and only tell the truth, mistakes in your answers can incriminate you (either by misspeaking or drawing simple conclusions).
• Even truthful answers can be contradicted by mistaken or unreliable evidence, destroying you credibility.

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

I found the second part of the lecture especially interesting, where a veteran detective (George Bruch) backs up Duane’s arguments. Definitely recommend this one to friends.

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

You can read about it HERE on the New York Times website.

I’m wondering how long until NASA applies for farm subsides?

A few facts about this video:

• The stuff falling from the rocket is ice. The Saturn engines burned liquid hydrogen and oxygen, which is VERY cold.
• The engines generate a strong vacuum, which is why the ice and smoke are initially sucked down into the opening in the launch pad base.
• The short section of dark exhaust coming out of the nozzles (right before the exhaust fire gets bright) is due to the cool liquid hydrogen that is used to cool the nozzles.

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

Being the nerd that I am, I then went to WebMD and looked up caffeine. Here’s somethings I found:

“Caffeine exaggerates the stress response,” says James D. Lane, PhD, professor of medical psychology at Duke University Medical Center in Durham, N.C., and a long-time caffeine researcher. “At the cellular level, caffeine locks the receptor normally used by adenosine, a brain modulator that provides feedback to avoid overstimulation of nerve cells. If adenosine is locked up, nothing keeps the nervous system from getting too excited at a cellular level.”

So what’s the harm, ask caffeine fans, who point to studies showing the benefits of caffeine, such as boosting memory and improving concentration and perhaps lowering risks of diseases such as Alzheimer’s and liver cancer.

But others are alarmed by what they say is an increasingly overcaffeinated nation; they are concerned by studies finding too much caffeine can set you up for high blood pressure, high blood sugar, and decreased bone density — not to mention jangled nerves.

And some fun facts from another article:

Children’s consumption of soft drinks has doubled in the past 35 years, with sodas supplanting milk.

By triggering the release of adrenaline to help muscles work harder and longer, caffeine so clearly enhances athletic performance that until 2004 it was considered a controlled substance by the International Olympic Committee.

The young adult crowd who favor caffeine with their alcohol appear to be putting themselves at some risk, too. According to Mark Fillmore, a psychologist at the University of Kentucky, “Caffeine seems to restore the speed of your behavior but not the accuracy.” This gives a whole new meaning to “The Quick and the Dead!”

…in Zimbabwe names are often chosen to convey specific meaning. Thus, one will find names such as Trymore, Lovemore, Learnmore, Justice, Honour, Trust, Knowledge, Oblivious, Wind, Wedding, Funeral, Rain, and even Hatred. Have-a-Look Dube is apparently a famous footballer in Zimbabwe. In one family, the last of 13 children was named “Never Trust A Woman” — apparently to express doubts about paternity.

A recent bill proposed in the Venezuala National Assembly would have effectively limited parents of newborns to a list of 100 names chosen by the government (the proposal failed to advance). The purpose? According to sponsors, the list was intended to “preserve the equilibrium and integral development of the child” by preventing parents from bestowing names that open the child to ridicule, are difficult to pronounce in Spanish, or generate doubts as to the child’s gender… Some Venezuelan names — e.g., Kennedy, John Wayne, Pavel, Ilich — reflect historical and cultural ties . In the voter registry, one can find as many as 60 Hitlers; eight Hochiminhs, among them Hochiminh Jesús Delgado Sierra; and six Eisenhowers, including Dwight Eisenhower Rojas Barboza.

Read the full post HERE.

The principle functions of a spacesuit are to create a pressurized, oxygenated atmosphere for astronauts, and to protect them from ultraviolet rays and extreme temperatures. Without it, a spacewalker would asphyxiate from the lack of breathable air and suffer from ebullism, in which a reduction in pressure causes the boiling point of bodily fluids to decrease below the body’s normal temperature. Since it takes a bit of time for these things to kill you, it’s possible to make it through a very quick stint in outer space.

At most, an astronaut without a suit would last about 15 seconds before losing conciousness from lack of oxygen.

You can read the full article here. There is also an interesting post about this on NASA’s Goddard Space Flight Center website:

How would the unprotected human body react to the vacuum of outer space? Would it inflate to bursting? or would it not? or would just the interior gases hyperinflate?…

answer:

You do not explode and your blood does not boil because of the containing effect of your skin and circulatory system. You do not instantly freeze because, although the space environment is typically very cold, heat does not transfer away from a body quickly. Loss of consciousness occurs only after the body has depleted the supply of oxygen in the blood. If your skin is exposed to direct sunlight without any protection from its intense ultraviolet radiation, you can get a very bad sunburn.

The full Goddard post has some interesting stories about past testing accidents involving loss of pressure.

For those outside the tech community:

RFC = Request For Comment

“foo”, “bar”, etc are used to quickly name things in a piece of software (usually when the name is not important, like when writing a quick piece of test code).

In other words, a dilatant material will become “thicker” or “harder” when it is hit. This is counter-intuitive… imagine a liquid that becomes solid when it is hit, then quickly returns to a liquid.

A simple example of a dilatant material is called “oobleck“. Oobleck is a mixture of cornstarch and water. Click here to learn how to make oobleck at home.

There is currently research going on that uses dilatant materials for flexible body armor. When a bullet strikes the fluid, it immediately becomes hard, protecting the person wearing it. The armor is full of polyethylene glycol with nano-particles of silica suspended in it. For you super-nerds: polyethylene glycol is a relative if ethylene glycol (anit-freeze), and is used to thicken shampoo and cosmetics.