Formant Shifting with Tympan

Once you have a real-time platform for manipulating audio, it is always fun to see what you can do to your voice.  In my case, I had been spending a lot of time figuring out a good way to implement frequency-domain audio processing on the Tympan.  Once I did that, I realized that it would be super easy to start having fun with my voice.  So, here, I present my Tympan Formant Shifter!

Formants vs Pitch.  Like any instrument, your voice has a pitch, which is the fundamental frequency of the sound of your voice.  The sound of your voice contains many harmonics, in addition to the fundamental frequency.  Those harmonics extend upward far above the fundamental frequency.  Which of those harmonics are actually projected from your mouth depend up how your mouth (and nasal passages and throat) are shaped.  As you talk, you naturally change the shape of your mouth (and nose and throat) to make the various consonant and vowel sounds.  An "E" sounds different from an "A" because your throat/nose/mouth enhance different harmonics for the two different vowels.  These enhanced frequency regions that move around -- these are your "formants".  

Formant Shifting.  In the video, I am only moving the formants.  Clearly, the effect on us listeners is that we feel like my voice itself is going higher or lower.  But, this isn't the case.  The fundamental frequency of my voice and the frequencies of the harmonics are all unchanged.  Instead, the formant shifting allows you to hear harmonics of my voice that are higher than your normally hear or lower than you normally hear.  The processing is shifting which harmonics are emphasized or attenuated.

Frequency-Domain Formant Shifter.  A formant shifter is implemented easily in the frequency domain.  Starting from your audio, take an FFT, shift the magnitude of the FFT bins to higher or lower bins (while leaving the FFT phases in their original bins), take the inverse FFT ("IFFT"), and play the audio.  In principle that's it!

Real World FFT/IFFT Processing.  In reality, of course, implementing the FFT/IFFT processing on a real-time audio stream is more complicated.  But, as I said at the top, I took quite a bit of time to try to hide all of these complications.  It takes care of the buffering, the windowing, and the overlap-and-add operations.

Tympan Example Code.  In the Tympan Library, I wrote an example sketch for Formant Shifting.  You can see it here.  The underlying audio processing class that does the formant shifting is here (*.h) and here (*.cpp).  Finally, for the video at the top, I combined the Formant Shifting with a USB Audio interface so that it would get recorded along with the video from my web camera.  You can get this USB Audio enabled version of the code (along with other partly-working goodies) here.

More Frequency-Domain Examples.  Once I got the whole frequency-domain processing structure in place, I found it fun and easy to implement several other frequency-domain algorithms.  In addition to Formant Shifting, I've got Frequency Shifting (but it is only linear shifting, not exponential scaling).  I've also got Frequency Compression and Noise Reduction.  I totally nerd-ed out.  So fun!

Measuring Audio Latency

For real-time audio processing, it is often important to minimize the delay between audio coming out of the system compared to the audio coming into the system.  This is especially true of hearing aids, where too much latency can cause the listener to perceive an echo (or have a comb-filtering effect), which end up degrading rather than helping the listener's experience.  Research suggests that the maximum tolerable latency in a hearing aid is only 14-30 msec.  If Tympan wants to be helpful, we need to make sure that we keep our latency shorter than this.  Let's find out what our system does!

Test Setup.  To measure the latency of the Tympan, we need to inject an audio signal into the Tympan and then measure delay of the output audio relative to the input audio.  There's lots of ways that this can be done.  I chose to use the setup shown in the picture above.  In this setup, I generate my test audio signals from my PC.  I use a series of 1 kHz tone bursts that are 1 second long.  The audio comes out of my PC and (as shown in the red line) is split so that it goes to the input of the Tympan *and* to the input of an audio recorder.  The output of the Tympan is then (as shown in the blue line) routed to the other input of the audio recorder.  The stereo audio file produced by the audio recorder will contain the input audio in the left channel and the output audio in the right channel.  It will then be a simple post-processing analysis to measure the delay between the two channels.

Raw Data.  An example recording and an example Matlab processing script are in my GitHub here.  The plot above shows the example data.  As you can see, there are three tone bursts in the recording.  At this timescale, one cannot see any delay between the signals, but that is because we are not zoomed in enough.  The plot below zooms in to the start of one of the tone bursts.  Here, we definitely see that the Tympan output has a slight delay relative to the direct audio signal.

Analysis.  Using the plot above, we could visually assess the latency between the two channels.  But, to do even better, I included a Matlab script in the GitHub directory that computes the cross-correlation between the two channels.  The best estimate of the latency will be when when the cross-correlation is maximum.  For this recording, the best estimate of the latency is 3.1 msec.  That's nice and short!

Measuring other Tympan Configurations.  Expanding from this single measurement, I then repeated the process and measured the latency for a variety of different configurations of the Tympan.  I tried different audio block sizes and I tried different audio processing algorithms.  My results are shown in the figure below.  The simple 3.1 msec value reported above can be seen in the bottom-left of the plot as the first point on the yellow line.  All of the other configurations result in increased latency, but there are still a lot of configurations that are shorter than the 14-30 msec upper limit from the literature.

Components of the Latency.  After working with the system for a while, I've identified three contributors to the latency:

• Hardware:  The audio interface for the Tympan is a TI 3206 AIC.  This chip has a pipeline that is 17 samples long on the input and 21 samples long for the output.  So, for this round-trip testing, it contributes 38 samples of latency, which at sample rate of 24 kHz is a latency of 1.58 msec.
• Audio Library:  The Tympan audio library is based on the Teensy audio library, which has employs a queue of two audio blocks in order to prevent audio hiccups and drop-outs.  In the Tympan library, this audio block size is configurable, hence my ability to make the graph above where I vary the block size.  For a block size of 16 samples, the library's latency is 2*16=32 samples, which is 1.33 msec at 24 kHz.  Totaled with the hardware latency (1.58 msec + 1.33 msec), we get 2.9 msec, which is very close to the 3.1 msec value that I measured.  Great!
• Audio Processing:  Beyond the audio library, most any actual audio processing will also introduce additional latency.  A typical (symmetric) FIR filter, for example, will result in a group delay that is half the length of the FIR filter.  Other filters and other operations (such as FFT) introduce delays as well.  The specific amount of latency may vary, but some latency is inherent in the math.  For the Tympan, we see the latency for FIR and FFT in the graph above.

Optimizing for Minimum Latency.  From this exploration, I've learned that latency can be minimized by (1) using the shortest audio block size that the system can handle and (2) running the simplest audio processing algorithm that you can get away with.  On this latter point, we all want our audio processing to have extremely fine frequency resolution.  But, high resolution requires long FIR filters or long FFTs.  Long FIRs/FFTs introduce a lot of latency, however.  So, if you want low latency, you need to use the shortest FIRs and FFTs that you can.

Benchmarking - Teensy 3.6 is Fast!

Fast but easy to program --  that's what I need for my audio processing projects.  In my previous post, I loved the speed of the NXP K66 processor, but was frustrated with the NXP development environment.   Well, thanks to PJRC and Kickstarter, the solution has arrived...via US Mail.  Welcome to the new Teensies!  Based on my tests, it looks like audio hacking just got a whole lot more fun.

Teensy?  Teensy is a line of microcontroller boards that are much more capable than the standard Arduino boards, but that can still be programmed using the friendly Arduino development environment (the "Arduino IDE").  I've used one of the existing Teensy boards (Teensy 3.2) on a number of projects and it definitely achieves its goal: it's a powerful, fast little board that is really easy to program.

New Teensies!  Through just-completed Kickstarter, Teensy has expanded its line of boards by adding the Teensy 3.5 and Teensy 3.6.  As can be seen in the table below, the Teensy 3.5 and 3.6 have a faster clock speed and have more RAM.  The specs on the Teensy 3.6 match by my loved-and-hated FRDM-K66F board from NXP.  The increased capabilities of the Teensy 3.5 and 3.6 relative to the older Teensy (and relative to Arduino) should really help in computationally-heavy tasks, such as processing audio.

Floating-Point Prowess.  For me, the best feature of the new Teensies is the inclusion of a floating-point unit (FPU), which will accelerate calculations with floating-point numbers.  This will makes it much easier to hack together audio processing algorithms.  I loved the performance of the NXP FRDM-K66 board because its FPU made it stunningly fast on floating-point FIR and FFT operations.  I'm hoping that the Teensy 3.6 is able to match the speed of the FRDM-K66.  If it can, I can leave behind the FRDM-K66 board and never again have to struggle with NXP's development environment.

Programming the New Teensies.  To evaluate the new Teensy boards, I'm going to re-run my FFT benchmark tests (on GitHub here).  First, I had to get the latest "Teensyduino" software from PJRC (to get support for the new boards into the Arduino IDE).  Then, I simply opened up my FFT benchmarking sketch and recompiled.  It was previously written for the Teensy 3.2 and now, without any changes, it compiled successfully for the Teensy 3.5/3.6.  This is the ease-of-use that I was hoping for!

Speed Results.  I ran my bechmarking tests on the new boards and complied the results in the table below.  It shows how many FFTs each board could complete per second.  It's my raw data.  It is hard to see the interesting trends in this table, so just go ahead and jump over it...I'll present something simpler in a moment...

Which Board is Fastest?  Below is a bar chart comparing the speed of each board for a 128-point FFT.  I shows the speed when using Int32 data types and when using Float32 data types (the two most likely used in audio processing).  Two results stand out:

• The Teensy 3.6 is the overall winner, even beating the FRDM-K66F.  While that is a bit inexplicable, it's great!
• The Teensy 3.5 and 3.6 are great for floating point calculations.  Look at how big those red bars are!  Clearly, the FPU in these newer boards makes a huge difference.

How Fast Do I Need?  So the bars in the graph above are big.  How do I know what bars are big enough?  How do I know which are fast enough for audio processing?  Well, like in my previous post, I want to know which boards can keep up with calculations that need to be done at audio rates.  In this case, I want to know which boards can do FFT operations at audio rates.  What does that mean?

FFTs for Audio Processing.  FFTs are interesting for audio because it means one can process audio in the frequency domain, which can be very convenient.  To do frequency-domain processing, you need to do an FFT to get into the frequency domain and you need to do an IFFT (which is basically the same computational load as an FFT) to get back out of the frequency domain.  Then, because of windowing and other blah-blah-blah, you generally overlap your audio data blocks by 50%, which means that each audio block needs four FFT-like operations.  Can this get done in real time?

Keeping up with Audio.  Because I measured how many FFTs each board can do per second, I can estimate the maximum audio sample rate (samples per second) that each board can support while still doing four FFTs per data block.  A typical sample rate used for audio is 44 kHz.  My results below show which boards can sustain at least 44 kHz (shown in green) versus those that cannot (shown in red).

Integer vs Floating-Point Results.  As can be seen in the table, the Teensy 3.2 is fast enough for audio if I'm using integer data types (Int16 or Int32).  It is not fast enough to use Floats.  The new Teensy 3.5 and 3.6 are faster than the Teensy 3.2, especially on Floats.  The Teensy 3.6 just screams on Floats -- it has enough processing power to sustain a sample rate over 300 kHz for frequency-domain processing.  Wow.  This is exactly the result that I was hoping to see.

Conclusion.  For my audio processing projects, it looks like the Teensy 3.6 is the best choice.  It's really fast, yet it can be easily programmed through the comfortable and friendly Arduino IDE.  I'm very pleased.

Next Step:  My next step is to join the Teensy 3.6 to an audio interface (such as the Teensy Audio Board) so that I can leave these synthetic benchmarks behind and start playing with actual audio.  It's gonna be fun!

Extra Credit:  How much faster are the new Teensies vs the Teensy 3.2?  The Teensy 3.6 is 16x faster than the Teensy 3.2 on Floats.  Dang, that's fast.  Again, this must be due to the FPU in the new units.

Follow-Up:  I made my own Teensy Hearing Aid using a Teensy 3.6 and a Teensy Audio Board.  It's fun!

Benchmarking - FFT Speed

My goal is to find a good microcontroller board for doing audio processing.  Speed is a very important concern so, in my last post, I looked at the speeds for different boards when doing FIR filters.  While time-domain FIR filters are an important audio processing task, I am also curious how suitable these boards are for frequency-domain processing.  In other words, I need to know how fast they can do FFTs.  Let's find out!

Why FFT?  An FFT is a "Fast Fourier Transform", which is not a helpful name if you're not already familiar with the idea.  FFTs are most often used when one wants to look at the frequency content (the spectrum) of audio data.  Intriguingly, you can also do an inverse FFT (an "IFFT") to convert that frequency spectrum data back into audio data.  By pairing the FFT with the IFFT, therefore, one can now manipulate (or mangle!) audio data in the frequency domain, which can be a more natural and easy way to construct one's audio processing algorithms.  They key to it all is the FFT (and its computationally similar IFFT).

Microcontroller Boards:  To understand which hardware might be capable of this frequency domain approach to audio processing, I'm evaluating a range of different boards.  In addition to spanning a range of clock speeds (16 MHz - 180 MHz), they also vary in other computationally important ways:  one is 8-bit while the others are 32-bit, some have DSP extensions while others do not, and one has a floating-point unit (FPU) while the others do not.  Though my testing, I'll see what's important.

My Test Software:  For all of the boards except for the FRDM-K66F, I used the Arduino IDE to write my test software.  It's a simple program that does an FFT on dummy data of a given length.  The program uses Arduino's "micros()" command to measure the time to complete a fixed number of FFTs.  Easy.  For the FRDM-K66F board, which cannot be programmed from the Arduino IDE, I had to use NXP's IDE (Kinetis Design Studio).  The FFT functions were identical, however, regardless of which IDE I used.  All of my software is available on my GitHub (for Arduino, for Kinetis).

KissFFT Function:  The only difficult part of the software is the FFT function itself.  Since I wanted to test across a variety of hardware, I wanted to start with an FFT routine that was written in generic C.  From the FFTW website, I found an interesting comparison of different FFT routines.  From their list, one of the most generic routines appeared to be the "KissFFT".  After downloading it, I refactored the KissFFT code to enable different data types (Int16 vs Int32 vs Float32) and to remove the dynamic allocation of memroy via malloc().  With these changes, it was much more suitable for use on a microcontroller.

Raw Results:  The raw results of my speed tests are here, from which I made the summary table shown below.  The table shows the number of FFTs per second that each board could complete.  I performed the tests for different data types (Int16, Int32, Float32) and for different FFT lengths (N=32 to N=512).  This table is too dense to read, so let's skip ahead.

Speed vs FFT Length:  First, let's look at the overall flow of the data.  The plot below shows the FFT speed for different FFT lengths.  For FFTs using more data points, I would expect slower performance.  This graph confirms that expectation.  Good.  This graph also shows that the relative ranking of the different boards is the same, regardless of the length of the FFT.  This allows me to greatly simplify the rest of the plots.

Speed for Integer Data:  As the length of the FFT doesn't matter for the relative rankings, I chose to focus on an FFT size of 128.  I chose this value because, when operating on audio data sampled at 44100 Hz, N=128 yields a frequency resolution of 344 Hz, which is a reasonably useful value.  The graph below compares the speeds of the different boards for doing 128-point FFT on Int32 data.  The Arduino Uno didn't have enough memory to complete this test, so it is excluded.  Otherwise, I see that the Arduino M0 is the slowest and that the FRDM-K66F is, by far, the fastest.  The only surprise in this data is the slowness of the Arduino M0.  It is much slower compared to the Teensy or Maple than is expected based on their relative clock speeds.  My data shows that Teensy can do ~4x the number of FFTs even though its clock speed is only ~2x higher.  Clearly the M0 is not optimized for these kinds of calculations.

Can They Do Floating Point?  For audio processing projects, I hope to do all of processing using floating point data types (Float32).  Being able to utilize floating point math (Float32) instead of fixed point math (Int16 and Int32) makes the algorithms much easier to design, debug, and optimize.  The difficulty is that microcontrollers tend to be very slow with floating point calculations.  So, let's do some tests with Floats and see if any of my boards are fast enough.

FFT Speed for Float32:  The graph below shows the FFT speeds that I measured using Float32 audio data.  As can be seen by the red bars, the M0, Maple, and Teensy are very slow on Floats.  They can only do 1/3rd as many Float32 FFTs and they can Int32 FFTs.  That is a major speed penalty.  The major exception to this trend is the FRDM-K66F.  It is actually faster using Float32 than Int32.  Presumably, this is due to the float-point unit (FPU) included in the K66F chip.  Even with the FPU, I did not expect it to be faster on Floats than Ints.  Very surprising.

DSP Acceleration:  Because an FFT is such a common digital signal processing (DSP) task, some processors include internal features to accelerate this kind of math.  The Teensy 3.2 and the FRDM-K66F are both based on the Arm Cortex M4 processor core.  The Cortex M4 includes DSP acceleration.  I learned that I can invoke the ARM's DSP accelerators by calling the FFT functions from ARM's "CMSIS" library.  It took me some effort to figure it out, but I eventually got it to work.  And, boy, did it work well...

FFT Speed Using CMSIS:  My results using the CMSIS library are shown below.  Using the CMSIS routines (and the underlying hardware acceleration) really speeds up the FFTs.

The effect of the CMSIS / DSP accelerators is so dramatic that I quantified the improvement in the table below.  The speed improvement is different for the different data types.  Using Int16 data, the CMSIS routines are 4-5x faster than the Generic C "KissFFT" routines.  Wow.  For the In32 data, CMSIS is 3x faster and for Float32 data, CMSIS is 2x faster.  This extra speed is definitely a good reward for the effort spent to figure out how to use the CMSIS library.

How Fast is Fast Enough?  I want to know which boards are fast enough to enable frequency-domain processing of audio signals.  As I discussed earlier, frequency-domain processing requires that I perform an FFT to get into the frequency domain and then an IFFT (which takes the same amount of time as an FFT) to get back out of the frequency domain.  Given my measured FFT speed values, I can estimate the maximum audio sample rate that each board can sustain:

max_sample_rate_Hz = (FFTs_per_second * N_FFT / 2) * (1 - overlap)

In this equation, the "2" accounts for the need to do both an FFT and an IFFT.  The "overlap" term accounts for the fact that most people do frequency-domain processing using blocks of audio samples that overlap by 50% (0.5) in order to smooth out any artifacts at the ends of the audio blocks.

Maximum Sample Rate for Frequency-Domain Processing:  Using this equation, assuming an N=128, and assuming an overlap of 0.5, the table below shows the maximum sample rate that each board can support for frequency-domain processing.  I've highlighted in green those boards that can sustain this kind of processing at sample rates appropriate for audio (ie, greater than 44100 Hz).

Conclusion:  If my goal is to do frequency-domain processing using Float32 data, only the FRDM-K66F is fast enough.  That is my primary conclusion.  My secondary conclusions are that, if I use Int32 data, the CMSIS acceleration means that the Teensy 3.2 is a viable option.  Furthermore, if I can tolerate Int16 data, I can choose from the Maple, Teensy, or K66.  But I don't want to do that.  I want to use Floats.  So, the K66F is for me.

Looking Forward:  While the FRDM-K66F board is very powerful for doing FFT operations, it is difficult for me to program.  I prefer the simplicity of the Arduino IDE, yet the FRDM-K66F is not supported by Ardiuno.  Looking forward however, the folks who do Teensy are about to release the "Teensy 3.6", which uses the same (or similar) processor as is used in the FRDM-K66F.  Since Teensy is programmable from the Arduino IDE, I am hoping that we'll soon have the power of the K66F combined with the ease-of-use of the Teensy.  That will be a truly winning combination for open source audio processing.