Speech & Audio
A speech model trained on 128 languages improved translation into English by an average of 7.4 BLEU over the previous best on the CoVoST-2 benchmark and cut speech recognition error rates by 14 to 34 percent relative to the best known prior work
In a 2022 paper published at Interspeech 2022, a peer-reviewed conference, researchers describe XLS-R, a self-supervised model for cross-lingual speech representation learning. They trained versions with up to 2 billion parameters on nearly half a million hours of publicly available speech audio in 128 languages. The authors say that is an order of magnitude more public speech data than the largest known prior work. On the CoVoST-2 speech translation benchmark, XLS-R improved translation into English by an average of 7.4 BLEU across 21 translation directions. BLEU is a score where higher means a translation is closer to a human reference translation.
What is XLS-R?
XLS-R is a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. In plain language, it learns patterns in spoken audio across many languages, which can then support tasks such as recognizing speech or translating it. The model is self-supervised, meaning it learns from raw speech audio without needing large amounts of human-written transcriptions.
How large was the training setup?
Researchers trained versions of XLS-R with up to 2 billion parameters, the adjustable values a model learns during training. Training used nearly half a million hours of publicly available speech audio in 128 languages. The authors describe that as an order of magnitude more public speech data than the largest known prior work.
What did the authors report for speech tasks?
For speech recognition, XLS-R improved over the best known prior work on BABEL, MLS, CommonVoice and VoxPopuli, lowering error rates by 14 to 34 percent relative on average. The authors also report that XLS-R set a new state of the art on VoxLingua107 language identification, the task of identifying which language is spoken.
What did the study show about cross-lingual pretraining?
The authors report that with enough model size, cross-lingual pretraining performed as well as English-only pretraining when translating English speech into other languages. They note that this setting usually favors English-only pretraining.
Limits and what's next
The abstract does not state limitations of the work. The authors hope XLS-R can help improve speech processing tasks for many more languages of the world.
Source: Arun Babu et al., “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,” Interspeech 2022, 2022.