Speech & Audio
Challenge results suggest spoof and deepfake speech detectors stayed accurate under new transmission effects but did not generalize well across real rooms and online speech datasets
In a 2023 paper in IEEE/ACM Transactions on Audio Speech and Language Processing, researchers reported results from the ASVspoof 2021 challenge, which compares systems that detect spoofed and deepfake speech. Fifty-four teams submitted systems to the evaluation phase. Detectors for the logical access (LA) task stayed accurate when new encoding and transmission effects were added. Detectors for the physical access (PA) task showed promise for catching replay attacks in real physical spaces but did not hold up when acoustic conditions changed from simulated to real. The new deepfake (DF) task looked at manipulated, compressed speech posted online. Those detectors showed some resistance to compression but did not generalize well across source datasets.
What did the challenge test?
The challenge ran three evaluation tasks. The logical access task introduced new encoding and transmission effects. The physical access task tested replay attacks in real and simulated physical spaces. The deepfake task was new in 2021 and targeted manipulated, compressed speech data posted online.
Which defenses held up across tasks?
The logical access detectors stayed accurate under the new effects. The physical access detectors showed potential in real spaces but lost accuracy when the acoustic setting differed from the simulated setting. The deepfake detectors coped somewhat with compression but did not transfer well across source datasets. The paper also summarizes the top-performing systems for each task and adds analyses of influential data factors and hidden data subsets.
FAQ
What is ASVspoof?
ASVspoof is a challenge series for spoofing and deepfake speech detection. The 2021 edition compared systems submitted by 54 teams to the evaluation phase.
How many teams took part?
Fifty-four teams submitted systems to the evaluation phase.
What was new in the 2021 edition?
The deepfake task was new. It targeted manipulated, compressed speech data posted online.
Limits and what's next
This is one challenge evaluation of the 54 teams that submitted systems to the evaluation phase. The tasks used logical access, physical access and deepfake audio, including simulated and real acoustic spaces and compressed online speech. The abstract reports no independent replication. The physical access results showed a lack of robustness to variations between simulated and real acoustic environments. The deepfake results showed a lack of generalization across source datasets. The abstract says the article also outlines the principal challenge limitations and a roadmap, but it does not spell them out.
Source: Xuechen Liu et al., “ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild,” IEEE/ACM Transactions on Audio Speech and Language Processing, 2023.