Speech & Audio
Models trained on 400,000 audio clips with ChatGPT-cleaned captions outperformed earlier audio-language models
In a 2024 paper in IEEE/ACM Transactions on Audio Speech and Language Processing, researchers describe WavCaps, a dataset of about 400,000 audio clips paired with written captions. Audio captioning is the task of describing what a sound clip contains in words. The team collected clips and rough text descriptions from web sources and from a sound event detection dataset. They used ChatGPT to filter and rewrite the noisy text into cleaner captions. Systems trained on WavCaps outperformed previous leading audio-language models by a significant margin on multiple downstream audio-language tasks, the authors report.
What is WavCaps?
WavCaps is a weakly labeled audio captioning dataset of about 400,000 audio clips with paired captions. The abstract does not define what weakly labeled means here. The clips and rough descriptions came from web sources and a sound event detection dataset.
How did ChatGPT help?
The authors used ChatGPT in a three-step process. It filtered noisy raw descriptions and rewrote them into cleaner captions. The abstract describes this as automatic filtering and transformation.
How did the trained systems perform?
The authors report that systems trained on WavCaps outperformed previous leading audio-language models by a significant margin on multiple downstream tasks. The abstract does not give separate scores for each task.
What else should a reader know?
What does weakly labeled mean?
The abstract does not define weakly labeled. The researchers started with existing noisy descriptions and used ChatGPT to make them cleaner.
Why did the researchers make WavCaps?
The abstract says earlier audio-language datasets were limited in size, and collecting audio captions is costly and time-consuming. The team aimed to provide a larger dataset and to show how large language models could support academic research.
Does the abstract give exact scores?
The abstract does not list exact scores for each downstream task. It reports a significant improvement over earlier leading models.
Limits and what's next
The results come from one study and one dataset, and the abstract reports no independent replication. The dataset was built from about 400,000 audio clips with rough descriptions from web sources and a sound event detection dataset. ChatGPT filtered and rewrote the captions, so the quality of the dataset depends on that automated step. The abstract does not say how the clips were selected or how diverse they were, and it does not give the exact size of the performance gap. The authors write that they hope WavCaps will help future audio-language multimodal research and show how large language models can support academic work.
Source: Xinhao Mei et al., “WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research,” IEEE/ACM Transactions on Audio Speech and Language Processing, 2024.