Speech & Audio
Nvidia's FastPitch can make text-to-speech more expressive when its pitch predictions change, without adding overhead
Researchers at Nvidia describe FastPitch in a 2021 paper in the IEEE International Conference on Acoustics Speech and Signal Processing. The model processes all parts of a sentence at once to turn text into speech. It also predicts the pitch contour, the rise and fall of the voice, while it works. When those pitch predictions change, the speech can become more expressive, better match the meaning of the sentence, and sound more engaging. The authors report that this pitch control improves the overall quality of the synthesized speech without adding computing overhead. For the intermediate sound representation of a typical utterance, the model runs at more than 900 times real time.
What does pitch prediction add to text-to-speech?
Pitch prediction gives the model a way to control intonation as it makes speech. Raising or lowering the predicted pitch uniformly creates speech that sounds like a person deliberately changing their voice. The authors report that changing those pitch predictions helps the speech match the meaning of the sentence and makes it more engaging for a listener.
How fast is FastPitch?
The model keeps a fully parallel Transformer design, which means it processes all parts of a sentence at once. The pitch conditioning does not add overhead. The paper reports a speed of more than 900 times real time for mel-spectrogram synthesis of a typical utterance. A mel-spectrogram is an intermediate sound representation that a speech model can turn into audio.
Common questions
Does predicting pitch slow the model down?
No. The paper reports no extra overhead from the pitch conditioning, and the model keeps a speed of more than 900 times real time for mel-spectrogram synthesis of a typical utterance.
Is the speech quality compared with other systems?
The abstract reports that conditioning on pitch contours makes the overall quality of the synthesized speech comparable to top existing systems. It does not give numerical comparison scores.
Can FastPitch change pitch while making speech?
Yes. The model predicts pitch contours during inference, and changing those predictions can alter intonation. That includes uniform increases or decreases that resemble voluntary voice modulation.
Limits and what's next
The abstract does not say whether the model was evaluated with human listeners. It does not state the dataset or benchmark used to train or evaluate the model. The speed result is for mel-spectrogram synthesis of a typical utterance, and the abstract gives no sample size or numerical quality score. The abstract does not mention independent replication, so these results rest on one study.
Source: Adrian Łańcucki, “Fastpitch: Parallel Text-to-Speech with Pitch Prediction,” IEEE International Conference on Acoustics Speech and Signal Processing, 2021.