VALL-E: The AI Model That Will Blow Your Mind (and Ears)
VALL-E generates speech from text using only a three-second audio sample. Microsoft's model combines neural networks and signal processing techniques to produce high-quality speech from a very small prompt.
How conventional TTS works, and where it struggles
Vocoders and acoustic models are widely used in cascading text-to-speech systems, with the mel spectrogram as the intermediate representation. Advanced systems can synthesise high-quality speech for a single speaker or a group of speakers, but they need clean, studio-grade recordings. The large volume of audio crawled from the internet does not meet that standard, which inevitably leads to poor performance. With small training sets, existing systems generalise badly.
Zero-shot TTS, meaning speech for a speaker the model has never heard, is genuinely tricky. Making a new speaker's synthetic voice sound both like them and natural is hard. Researchers developed speaker adaptation and speaker encoding techniques to help, but these usually require fine-tuning, hand-designed features, or substantial changes to the network structure.
The alternative: more data
We have seen enormous improvements in text generation by training models on huge, diverse corpora. The same applies to speech. Rather than building an intricate network specialised for zero-shot TTS, train on as much data as you can get. The field went from around 16GB of uncompressed text to as much as 1TB.
Think of it like baking. Instead of a fancy multi-layer pan with intricate designs, sometimes the right move is a big mixing bowl and as many ingredients as you can find. The more diverse and plentiful the ingredients, the better the result.
The team also used a significant amount of semi-supervised data to build a generic system across the speaker dimension, which highlights how underused semi-supervised data has been in TTS.
Results
VALL-E synthesises realistic speech with high speaker similarity in the zero-shot setting, and it produces varied outputs from the same input text while preserving the acoustic environment and the speaker's mood from the prompt. Evaluated against the strongest prior zero-shot systems on LibriSpeech and VCTK, it came out ahead. The researchers describe it as the first TTS framework with robust in-context learning capabilities comparable to GPT-3.
In plainer terms, VALL-E uses a short audio sample to learn the patterns in someone's speech, then generates new speech matching those patterns. The output is close to indistinguishable from real human speech, which is a significant breakthrough and opens up applications from personal assistants to speech-based interfaces.
It also opens up an obvious risk surface, which is worth sitting with. A voice is now clonable from a three-second sample.