Speech & Audio
A speech model pre-trained on unlabeled audio improved phoneme classification, speaker recognition, and spoken sentiment classification
In a 2020 paper from researchers at National Taiwan University, a speech model called Mockingjay improved performance on phoneme classification, speaker recognition, and sentiment classification of spoken content. The model is pre-trained on unlabeled speech by looking at the sounds before and the sounds after each moment. That learned representation helped separate classification models perform better. In a low-resource test, Mockingjay used only 0.1 percent of the labeled data and still outperformed Mel-features, a standard audio feature method, which used all 100 percent of the labeled data. The paper appeared in the IEEE International Conference on Acoustics Speech and Signal Processing.
How does Mockingjay learn from unlabeled speech?
Mockingjay uses bidirectional Transformer encoders, a type of neural network that reads sequences from both directions. Instead of only using past sounds to predict future sounds, it predicts the current sound using both past and future context together. This pre-training on unlabeled speech creates a representation that later tasks can use.
What did the paper find across tasks?
The representation improved performance on phoneme classification, the task of identifying basic speech sounds, speaker recognition, and sentiment classification of spoken content. The authors also fine-tuned the model on downstream tasks. With only two epochs, or passes through the training data, they reported a large further improvement. In the low-resource setting with 0.1 percent of labeled data, Mockingjay outperformed Mel-features trained on all of the labeled data.
What are the common questions about this paper?
What is a transformer encoder?
A transformer encoder is a neural network component that processes a sequence by weighing many positions at once. In Mockingjay, it weighs both past and future speech frames.
What does the low-data result mean?
It means that in a low-resource setting with only 0.1 percent of the labeled data, Mockingjay outperformed Mel-features, which used all 100 percent of the labeled data.
Did the paper compare Mockingjay with other methods?
Yes. The abstract says Mockingjay outperformed other approaches on the tasks the researchers tested. It does not name those approaches in the abstract.
Limits and what's next
The abstract describes one study from researchers at National Taiwan University. It reports no independent replication. The abstract does not name the unlabeled speech dataset or its exact size, only that the model was pre-trained on a large amount of unlabeled speech. The improvements are reported on the downstream tasks the authors selected. The abstract does not describe a next step.
Source: Andy T. Liu et al., “Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders,” IEEE International Conference on Acoustics Speech and Signal Processing, 2020.