Formant Shifting with Tympan

Once you have a real-time platform for manipulating audio, it is always fun to see what you can do to your voice.  In my case, I had been spending a lot of time figuring out a good way to implement frequency-domain audio processing on the Tympan.  Once I did that, I realized that it would be super easy to start having fun with my voice.  So, here, I present my Tympan Formant Shifter!

Formants vs Pitch.  Like any instrument, your voice has a pitch, which is the fundamental frequency of the sound of your voice.  The sound of your voice contains many harmonics, in addition to the fundamental frequency.  Those harmonics extend upward far above the fundamental frequency.  Which of those harmonics are actually projected from your mouth depend up how your mouth (and nasal passages and throat) are shaped.  As you talk, you naturally change the shape of your mouth (and nose and throat) to make the various consonant and vowel sounds.  An "E" sounds different from an "A" because your throat/nose/mouth enhance different harmonics for the two different vowels.  These enhanced frequency regions that move around -- these are your "formants".  

Formant Shifting.  In the video, I am only moving the formants.  Clearly, the effect on us listeners is that we feel like my voice itself is going higher or lower.  But, this isn't the case.  The fundamental frequency of my voice and the frequencies of the harmonics are all unchanged.  Instead, the formant shifting allows you to hear harmonics of my voice that are higher than your normally hear or lower than you normally hear.  The processing is shifting which harmonics are emphasized or attenuated.

Frequency-Domain Formant Shifter.  A formant shifter is implemented easily in the frequency domain.  Starting from your audio, take an FFT, shift the magnitude of the FFT bins to higher or lower bins (while leaving the FFT phases in their original bins), take the inverse FFT ("IFFT"), and play the audio.  In principle that's it!

Real World FFT/IFFT Processing.  In reality, of course, implementing the FFT/IFFT processing on a real-time audio stream is more complicated.  But, as I said at the top, I took quite a bit of time to try to hide all of these complications.  It takes care of the buffering, the windowing, and the overlap-and-add operations.

Tympan Example Code.  In the Tympan Library, I wrote an example sketch for Formant Shifting.  You can see it here.  The underlying audio processing class that does the formant shifting is here (*.h) and here (*.cpp).  Finally, for the video at the top, I combined the Formant Shifting with a USB Audio interface so that it would get recorded along with the video from my web camera.  You can get this USB Audio enabled version of the code (along with other partly-working goodies) here.

More Frequency-Domain Examples.  Once I got the whole frequency-domain processing structure in place, I found it fun and easy to implement several other frequency-domain algorithms.  In addition to Formant Shifting, I've got Frequency Shifting (but it is only linear shifting, not exponential scaling).  I've also got Frequency Compression and Noise Reduction.  I totally nerd-ed out.  So fun!

A Teensy Hearing Aid

Hearing aids are totally closed devices -- their inner workings are hidden.  Access is limited to only those who work for the hearing aid companies.  But if innovation is to accelerate, we need more ideas iterated more quickly.  We need more people to participate in hearing aid development.  But if the devices are closed, there is no way to try new ideas.  So, let's consider the alternative.  Let's try to build an open-source hearing aid.  Yes, at first, an open-source hearing aid will be absurdly big and ugly.  But, you have to start somewhere.  I'm going to start here: take one Teensy microcontroller, add some supporting electronics, and VOILA! A Teensy Hearing Aid!

Basic Hearing Aid Elements:  While it is unfortunate that they are closed devices, real hearing aids are absolutely amazing pieces of technology.  Their minuscule packages are absolutely stuffed with functionality.  I, though, am going to start more simply.  I will start with some microphones, analog and digital converts, a digital audio processor, some speakers, and a battery.  Once this works, I can always add more features later.

Choosing a Processor:  For me, the biggest challenge is always with the software.  Therefore, I need to choose a digital audio processor that it is easy to program.  For me, that means choosing a processor that can be programmed from the hobbyist-friendly Arduino programming environment.  Within that universe of processors, I've chosen to use the Teensy 3.6 because it's fast and because it has a nice audio processing library to make it even easier to program for audio.

Supporting Elements:  To get audio signals into and out of the processor, the Teensy folks offer an inexpensive Audio Adapter Board that mates directly to the Teensy 3.6.  It has the audio ADC and DAC that I need.  For the microphones, I'm using a pair of mic breakout boards from Adafruit.  For speakers, I'm using whatever headphones or earbuds that I might have on-hand.  Finally, for the battery, I'm using a Li-Po battery and Li-Po charger from Adafruit.

Wiring It Up:  The figure above gives an overview of how everything was connected together.  As you can see, the Audio Board is at the center with everything else connecting to it.  The only tricky part of this setup is connecting the battery and charger into the Teensy system.  As indicated on the Teensy pinout diagram, you have to cut a trace on the backside of the Teensy so that you can insert the battery connections.  Once cut, the Teensy's 5V USB voltage goes to the battery charger and the battery's output goes back to the Teensy's "5V" input pin.  Not too bad.

Initial Software:  While this post is primarily about the hardware, I did write some basic software in order to see if the hardware is working (see my GitHub repo).  I used the Teensy Audio Library to configure the Audio Board to pass the "I2S" inputs (ie, my microphones) to the "I2S" outputs (ie, my headphones).  In between, I wrote a simple routine that applies a user-controllable amount of gain to make things louder.  I use the blue potentiometer on the Audio Board to set the amount of gain.

First Audio:  Of course, the first time that I tried to compile the code, my software didn't work.  Does anyone's code ever work the first time?  After some iteration, I finally got it to compile and upload.  Putting on my headphones, I could hear the audio being picked up by the microphones.  Turning the blue pot, I could control its volume.  Hardware knobs are so satisfying.  My favorite part, though, is being able to use the on-board battery so that I can move around freely.  Very fun.

Limitations:  Sure, my home-brewed "hearing aid" is ridiculously large -- no one (including me) would ever wear this around in everyday life.  But, unlike commercially-available hearing aids, my device is open.  Anyone can modify it to make it better.  Anyone can try out their own audio processing algorithms to try to improve one's hearing.  Will any of us beat the professional hearing aid algorithm designers?  Probably not.  But, for me at least, I will surely learn a lot by trying a few of the standard approaches.  And, I'll get to look really cool sharing pictures of myself with this (not so) Teensy Hearing Aid.  Happy hacking!

Follow-Up:  I measured the power consumption of my Teensy Hearing Aid.  Using the "WFI" command, I found a super-easy way to increase the battery life by 50%.  Wow!  See my post here.

Follow-Up:  I measured how the volume control affects the headphone ouput.  See here.