Speech & Audio
A neural codec language model can synthesize high-quality personalized speech from a 3-second recording, peer-reviewed study reports
Researchers trained a neural codec language model called VALL-E to treat text-to-speech as a language prediction task over discrete speech codes, rather than as continuous signal regression. The training set was 50,000 hours of English speech. The authors report that with only a 3-second recording of an unseen speaker as a prompt, VALL-E produced high-quality personalized speech. In the paper's experiments, the system significantly outperformed the state-of-the-art zero-shot text-to-speech system on speech naturalness and speaker similarity. The abstract also says VALL-E preserved the speaker's emotion and acoustic environment from the prompt. Microsoft, a company on the BDS divestment list, is among the authors' institutions.
What is a neural codec language model?
A neural codec language model is a language model that works with discrete speech codes produced by an audio codec. In this study, VALL-E treated text-to-speech synthesis as a conditional language modeling task.
How did the researchers test it?
The researchers used 50,000 hours of English speech for pre-training. For zero-shot synthesis, they gave the system a 3-second recording of an unseen speaker as a prompt. The abstract does not report the number of speakers or test samples in the evaluation.
What did the evaluation show?
The authors report that VALL-E significantly outperformed the state-of-the-art zero-shot text-to-speech system on speech naturalness and speaker similarity. According to the abstract, the system could also preserve the speaker's emotion and acoustic environment from the prompt in synthesis.
Frequently asked questions
Does the system need a long voice sample?
No. The paper reports that it used only a 3-second recording of an unseen speaker as a prompt.
What does zero-shot mean here?
The abstract does not define zero-shot.
What can the model preserve from the prompt?
The abstract says it preserved the speaker's emotion and acoustic environment in synthesis.
Limits and what's next
The abstract does not state limitations or next steps. The paper is peer-reviewed and appeared in IEEE Transactions on Audio Speech and Language Processing on 2025-01-01. The source does not list it as a preprint.
Source: "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers," IEEE Transactions on Audio Speech and Language Processing, 2025-01-01.