Measuring Audio Latency

For real-time audio processing, it is often important to minimize the delay between audio coming out of the system compared to the audio coming into the system.  This is especially true of hearing aids, where too much latency can cause the listener to perceive an echo (or have a comb-filtering effect), which end up degrading rather than helping the listener's experience.  Research suggests that the maximum tolerable latency in a hearing aid is only 14-30 msec.  If Tympan wants to be helpful, we need to make sure that we keep our latency shorter than this.  Let's find out what our system does!

Test Setup.  To measure the latency of the Tympan, we need to inject an audio signal into the Tympan and then measure delay of the output audio relative to the input audio.  There's lots of ways that this can be done.  I chose to use the setup shown in the picture above.  In this setup, I generate my test audio signals from my PC.  I use a series of 1 kHz tone bursts that are 1 second long.  The audio comes out of my PC and (as shown in the red line) is split so that it goes to the input of the Tympan *and* to the input of an audio recorder.  The output of the Tympan is then (as shown in the blue line) routed to the other input of the audio recorder.  The stereo audio file produced by the audio recorder will contain the input audio in the left channel and the output audio in the right channel.  It will then be a simple post-processing analysis to measure the delay between the two channels.

Raw Data.  An example recording and an example Matlab processing script are in my GitHub here.  The plot above shows the example data.  As you can see, there are three tone bursts in the recording.  At this timescale, one cannot see any delay between the signals, but that is because we are not zoomed in enough.  The plot below zooms in to the start of one of the tone bursts.  Here, we definitely see that the Tympan output has a slight delay relative to the direct audio signal.

Analysis.  Using the plot above, we could visually assess the latency between the two channels.  But, to do even better, I included a Matlab script in the GitHub directory that computes the cross-correlation between the two channels.  The best estimate of the latency will be when when the cross-correlation is maximum.  For this recording, the best estimate of the latency is 3.1 msec.  That's nice and short!

Measuring other Tympan Configurations.  Expanding from this single measurement, I then repeated the process and measured the latency for a variety of different configurations of the Tympan.  I tried different audio block sizes and I tried different audio processing algorithms.  My results are shown in the figure below.  The simple 3.1 msec value reported above can be seen in the bottom-left of the plot as the first point on the yellow line.  All of the other configurations result in increased latency, but there are still a lot of configurations that are shorter than the 14-30 msec upper limit from the literature.

Components of the Latency.  After working with the system for a while, I've identified three contributors to the latency:

• Hardware:  The audio interface for the Tympan is a TI 3206 AIC.  This chip has a pipeline that is 17 samples long on the input and 21 samples long for the output.  So, for this round-trip testing, it contributes 38 samples of latency, which at sample rate of 24 kHz is a latency of 1.58 msec.
• Audio Library:  The Tympan audio library is based on the Teensy audio library, which has employs a queue of two audio blocks in order to prevent audio hiccups and drop-outs.  In the Tympan library, this audio block size is configurable, hence my ability to make the graph above where I vary the block size.  For a block size of 16 samples, the library's latency is 2*16=32 samples, which is 1.33 msec at 24 kHz.  Totaled with the hardware latency (1.58 msec + 1.33 msec), we get 2.9 msec, which is very close to the 3.1 msec value that I measured.  Great!
• Audio Processing:  Beyond the audio library, most any actual audio processing will also introduce additional latency.  A typical (symmetric) FIR filter, for example, will result in a group delay that is half the length of the FIR filter.  Other filters and other operations (such as FFT) introduce delays as well.  The specific amount of latency may vary, but some latency is inherent in the math.  For the Tympan, we see the latency for FIR and FFT in the graph above.

Optimizing for Minimum Latency.  From this exploration, I've learned that latency can be minimized by (1) using the shortest audio block size that the system can handle and (2) running the simplest audio processing algorithm that you can get away with.  On this latter point, we all want our audio processing to have extremely fine frequency resolution.  But, high resolution requires long FIR filters or long FFTs.  Long FIRs/FFTs introduce a lot of latency, however.  So, if you want low latency, you need to use the shortest FIRs and FFTs that you can.

Benchmarking - FIR Filtering

I want to do real-time audio processing.  And, I want to do it using small electronics without having to fight with an operating system like Windows or Linux.  Most of my previous experience with electronics hacking has been with Arduino (or Arduino-like) platforms.  Audio processing, however, is pretty computationally demanding.  I need to find an easy-to-use board that is fast for typical audio tasks.  So, as a first step, I'm going to get a bunch of different boards and see which can do FIR filtering at audio rates.  Let's see which boards are up to the task!

Top: Arduino Uno, Arduino M0, LeafLabs Maple.  Bottom: Teensy 3.2 and FRDM-K66F.

The Competitors:  I chose to test six different boards -- many of which I already had kicking around the house.  The six boards that I tested are summarized in the table below.  As you can see, I tried everything from the lowly Arduino Uno up to the mighty Teensy 3.2 and the even-mightier FRDM-K66F.  While the FRDM-K66F is a bit obscure, I'm using it as a proxy for the up-coming Teensy 3.6, which uses the same K66F chip.  Its fast 180 MHz clock speed and its floating point unit (FPU) should make the FRDM-K66F / Teensy 3.6 great for processing audio.

Why FIR Filters?  If you want to manipulate the frequency content of an audio stream, you need a filter.  Boosting the bass?  Cutting some mids?  Boosting the treble?  Apply the appropriate filter.  There are many kinds of filters, typically divided into either IIR filters or FIR filters.  I'm not yet ready to dive into the differences between IIR vs FIR, but I tend to prefer FIR filters for their linear phase and unconditional stability (a good discussion of FIR filters is here and here).  Regardless, FIR filters are a good, basic audio processing task that make for a widely-applicable benchmark.

Lots of Multiplies and Adds:  The challenge with FIR filters is that they can require a processor to do a lot of computation -- a lot of multiply and addition operations.  The finer the frequency resolution desired, the more multiplies and adds are needed to do the filter.   The resolution of an FIR filter scales with the length of the filter (the "N" of the filter).  As a simple rule-of-thumb, an FIR filter's workload scales as:

N_FIR = sample_rate_Hz / freq_res_Hz;      //approximate
num_multiplies = N_FIR * sample_rate_Hz;   //same for num_adds

For example, if you want a frequency resolution of 250 Hz, and if your sample rate is 44 kHz, then you need an FIR filter length N = (44000/250) =  176.  To actually filter the audio, you need to apply this 176-point filter to every audio sample in your 44 kHz audio stream.  To keep up, your processor will need to do at least (176*44000) = 7.7 million multiplications plus 7.7 million additions per second.  That's a lot of work to do!  Which of my boards are capable of this?

FIR Software:  To test the FIR speed of each board, I used the Arduino IDE and wrote a very naive implementation of an FIR filter (yes, faster results could definitely be achieved, but this test is just trying to get a sense of relative speeds of the platforms).  My code uses the Arduino's "micros()" command to measure the time to repeat the FIR filter numerous times.  For the K66 board, which could not be programmed through the Arduino IDE (until the Teensy 3.6 comes out!), I had to use NXP's "Kinetis Design Studio" to write the software, but the FIR function itself is the same.  All of the code is available in my OpenAudio repository on GitHub.

Results, All Data:  My raw data (here) consist of the time required to perform FIR filters on the different platforms.  To ease the presentation of the data, I invert the values so that it tells me the number of FIR filters that can be completed per second.  In this perspective, a bigger number means it can do more FIRs per second, which is good.  My results are shown in the table below.  It shows the FIR speeds for different filter lengths (16-256) and for two different data types (Int32 and Float32).  Because I find tables difficult to read, let's jump over the table and make some plots that better illustrate the results.

Results, Effect of "N":  Longer filters require more computations, so I would expect longer filters to be slower.  Using data from the big table above, the plot below confirms that expectation.  Also, note that all of the lines show the same slope and that the lines never cross each other.  This means that the relative ranking of the different boards stays the same across all FIR filter lengths, which allows me to greatly simplify the rest of the plots.

Results, Speed of Each Board (Int32):  Since the relative ranking of the boards stays the same throughout, let's illustrate the relative speed of each board by picking just one filter length.  The plot below picks N=128.  It shows the speed when using Int32 data.  On the left side of the plot, note that the Arduino Uno is very slow on Int32 values.  Presumably its 8-bit processor has difficulty with the 32-bit data type.  On the right side of the plot, the fastest board is the K66F, which is 100 times faster than the Uno.  That's a huge difference!

Results, Floating Point:  The below below is the same, but I add in the results for floating point data (Float32).  Writing audio processing algorithms using Float operations is much easier than using Int operations, so these are the results that I'm most interested in.  As can be seen in the plot, these boards are *much* slower on Floats than on Ints.  The major exception to this result is the K66F board, which is basically as fast on Floats as it is on Ints.  That's amazing!  This result clearly reflects the fact that the K66 is the only processor in this comparison which has an FPU.

How Fast is Fast Enough?  Deciding what FIR speed is "fast enough" for audio processing is not a simple question.  One approach is to return back to the example at the top of this post: a hypothetical 176-point FIR filter.  This filter length was chosen because it would yield a frequency resolution of 250 Hz when run at an audio sample rate of 44 kHz.  Which of these boards can support such a long filter at this fast sample rate?  Well, by scaling the N=128 speed values to N=176, the table below shows the sample rate that could be sustained by each board.  As can be seen the Teensy 3.2 is fast enough for audio processing using Ints.  But, if I want to do Floats, only the K66F is fast enough.

Programming the K66F:  Unfortunately, the FRDM-K66F board is not programmable from the Arduino IDE.  This makes it much harder to setup and debug by non-professionals.  This a real hurdle.  Luckily, the upcoming Teensy 3.6 also uses the K66 processor.  Since the Teensy products are all compatible with the Arduino IDE, it means that the power of the K66 will soon be far more accessible.  That's the solution that I'm really looking for.  So, I supported the Teensy 3.6 kickstarter.  I can't wait to get my Teensy 3.6!

Next Steps:  FIR filtering is not the only audio processing task that one might like to do.  An FFT is another important type of operation that is computationally intense.  In my next post, I'll look at the FFT speeds to see which boards are capable of real-time, frequency-domain processing.  Until then, have some happy hacking!

Update: FFT Benchmarking Results are here!