Speech & Audio
Llama-AVSR reports a 0.77 percent word error rate on the LRS3 audio-visual speech benchmark
Researchers at Imperial College London, Meta, and Fondazione Bruno Kessler describe a model called Llama-AVSR that turns speech into text using sound and video of the speaker's lips. In a 2025 paper at the IEEE International Conference on Acoustics Speech and Signal Processing, they report a word error rate (WER) of 0.77 percent for audio-visual speech recognition and 0.79 percent for audio-only speech recognition on LRS3, a public benchmark. WER counts how many words the system gets wrong, so a lower number means fewer mistakes.
How did the model work?
The authors paired a pre-trained large language model with pre-trained audio and video encoders. The encoders turn sound and lip movements into tokens, the small units a language model reads, alongside text tokens. Only small connector parts and LoRA adaptation modules were trained. The main encoders and the language model, a Llama3.1-8B model, stayed frozen, so the system updated only a small number of trainable parameters.
Why did the researchers use lip movements?
The authors note that lip movements carry information that does not change with background noise.
What did the study find?
The paper reports new state-of-the-art results for automatic speech recognition and audio-visual speech recognition on LRS3. The authors write that they also studied which pre-trained encoders and language model choices help the model work well, and how much the audio and visual tokens can be compressed to balance accuracy and computing cost.
FAQ
What is a word error rate?
Word error rate is the share of words a speech recognizer gets wrong. A WER of 0.77 percent means fewer than one word in a hundred was wrong on the benchmark.
What is LRS3?
LRS3 is a public audio-visual speech recognition benchmark. The authors call it the largest public audio-visual speech recognition benchmark.
Who did this research?
Researchers at Imperial College London, Meta, and Fondazione Bruno Kessler did the work. Meta is one of the affiliations named on the paper.
Limits and what's next
This is one study, and the model was evaluated on one public benchmark, LRS3. The abstract reports no independent replication on another dataset. The paper appeared in the IEEE International Conference on Acoustics Speech and Signal Processing, but the abstract does not say how the model would perform on speech outside the benchmark.
Source: Umberto Cappellazzo et al., “Large Language Models are Strong Audio-Visual Speech Recognition Learners,” IEEE International Conference on Acoustics Speech and Signal Processing, 2025.