Speech & Audio
TERA, a self-supervised speech model, outperformed earlier models on several speech tasks in a large-scale comparison
In a 2021 paper, researchers at National Taiwan University and Amazon report that TERA, a self-supervised speech model, improved on surface features, the acoustic features used as a baseline, and outperformed earlier models. The work appeared in IEEE/ACM Transactions on Audio Speech and Language Processing. The team tested the model on phoneme classification, keyword spotting, speaker recognition, and speech recognition. TERA stands for Transformer Encoder Representations from Alteration, and it learns by reconstructing acoustic frames from altered versions of the audio along time, frequency, and magnitude.
What did the researchers build?
They built TERA, a self-supervised speech pre-training method. Self-supervised here means the model learned from a large amount of unlabeled speech rather than from labels supplied by people. TERA trains Transformer encoders, a type of neural network for processing sequences, by changing audio along three independent axes: time, frequency, and magnitude. A random policy decides the alterations, and the model then reconstructs the original acoustic frames from the altered version. The method can be used to extract speech representations or to fine-tune a model for a specific downstream task.
How well did TERA perform?
The paper presents a large-scale comparison of self-supervised models. In that comparison, TERA improved on surface features and outperformed previous models. The researchers also studied different alteration techniques, more pre-training data, and pre-training on various features. They report that the method transfers to downstream datasets that were not used in pre-training.
What do the model-size findings show?
The authors compared different model sizes. Smaller models were stronger at extracting speech representations, while larger models were more effective for downstream fine-tuning, which means adapting a pre-trained model to a specific task.
What questions does the paper leave open?
Does TERA transfer to new data?
The authors report that the method is transferable to downstream datasets not used in pre-training. The abstract does not name those datasets.
What were the smaller models better at?
Smaller models were stronger at learning speech representations, while larger models were more effective for downstream fine-tuning.
What tasks did the paper test?
The paper evaluated TERA on phoneme classification, keyword spotting, speaker recognition, and speech recognition.
Limits and what's next
The study evaluated TERA on phoneme classification, keyword spotting, speaker recognition, and speech recognition. The abstract does not name the datasets or the amount of unlabeled speech used for pre-training. This is one paper, and the abstract reports no independent replication. The abstract reports a comparison of several self-supervised models but does not list the previous models or the exact scores. The abstract does not say what follow-up work the authors plan.
Source: Andy T. Liu et al., “TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech,” IEEE/ACM Transactions on Audio Speech and Language Processing, 2021.