Speech & Audio
A self-supervised speech emotion model outperformed both general and specialist recognizers after only linear-layer training
In a 2024 paper, researchers describe emotion2vec, a speech emotion representation model. The authors pre-trained the model on open-source unlabeled emotion data and then tested it on a mainstream speech emotion recognition dataset called IEMOCAP. They report that, after only linear-layer training, emotion2vec outperformed both pre-trained general models and emotion specialist models that the authors describe as state-of-the-art. The same model also showed consistent improvements across speech emotion datasets in 10 languages and positive results on song emotion recognition, emotion prediction in conversation, and sentiment analysis.
How was emotion2vec trained?
The researchers used self-supervised pre-training, which means the model learned from patterns in the data rather than from human labels. The training data were open-source and unlabeled. During pre-training, the model combined two training signals. One signal considered whole utterances, meaning spoken segments. The other considered shorter frames within an utterance, meaning small time slices.
What did the tests show?
On the IEMOCAP dataset, a mainstream speech emotion recognition benchmark, emotion2vec outperformed pre-trained general models and emotion specialist models after only linear-layer training. A linear layer is a simple input-to-output mapping. The researchers also report consistent improvements across speech emotion recognition datasets in 10 languages. On other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis, the authors report positive results.
What evidence supports the claim?
The paper includes comparison experiments, ablation experiments, and visualization. Ablation experiments test what happens when parts of a model are removed or changed. The authors say these experiments support the model's general capability across emotion-related tasks.
What else should readers know?
Background on speech emotion representation models
The abstract does not define what a speech emotion representation model is. As general background, a model of this kind can turn speech into a numerical summary that captures emotion-related information. Other models can use that summary for tasks such as recognizing emotion from speech.
What is self-supervised pre-training?
It is an initial learning stage in which a model learns from patterns in unlabeled data rather than from labels provided by people.
Did the model need retraining for each task?
For the main speech emotion recognition test, the researchers trained only linear layers on the pre-trained model.
Limits and what's next
The abstract does not state limitations.
Source: Ziyang Ma et al., “emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation,” 2024.