Speech & Audio
A text-free voice conversion method outperformed the latest models trained with annotated data

Voice conversion changes a recording so the words stay the same but the voice sounds like a different speaker. In a 2023 paper, researchers at Wuhan University describe a method that learns this conversion without text labels. The paper appeared in the IEEE International Conference on Acoustics Speech and Signal Processing. In their experiments, the method outperformed the latest voice conversion models trained with annotated data and showed greater robustness.
Why do existing voice conversion methods struggle?
Current approaches often extract content that still has speaker information mixed in, or they need large amounts of annotated data. A mismatch between the conversion model and the vocoder, the part that turns the model's internal representation back into sound, can also lower the quality of the reconstructed waveform.
How does the new method work?
The researchers used an end-to-end VITS model, which goes from speech input to a reconstructed waveform. They separated content from speaker details by applying an information bottleneck to speech features from a model called WavLM. The bottleneck limits how much detail passes through, which helps keep what is said separate from who is speaking. They also used a data augmentation step that resizes spectrograms, the visual maps of sound frequencies, to make the extracted content information cleaner. This design did not need text annotation.
What are some common questions about this work?
How did it perform?
In their experiments, the researchers reported that the method outperformed the latest voice conversion models trained with annotated data and showed greater robustness.
What does one-shot mean?
The abstract does not say what one-shot means in this work. The term appears in the paper's title.
What did the method avoid?
The method avoided text labels by learning cleaner content directly from speech features. The method extracts content information without text annotation, unlike approaches that demand large amounts of annotated data.
Limits and what's next
The experiments compared the method with the latest voice conversion models trained with annotated data, but the abstract does not specify the dataset, the number of speakers, or the recording conditions. This is one study, and the abstract reports no independent replication. The abstract also does not say what the authors plan to test next. Independent testing would be needed to confirm that the reported advantage holds outside the described experiments.
Source: Jingyi Li et al., “Freevc: Towards High-Quality Text-Free One-Shot Voice Conversion,” IEEE International Conference on Acoustics Speech and Signal Processing, 2023.