Biased Self-supervised learning for ASR
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 4 Nov 2022.
Builds on
19 of 29 references.
References · 19
- 2017Attention Is All You Needcited by 60,106
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
- 2016Understanding deep learning requires rethinking generalizationcited by 1,666
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,625
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,359
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 337
- 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
- 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
7 more
- 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 140
- 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
- 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
- 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
- 2020Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Datacited by 14
- 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
- 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Modelscited by 10
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1.
2 papers.