mlsanityData to 14 Oct 2026

Training Autoregressive Speech Recognition Models with Limited in-domain Supervision

arXiv 2210.15135 ↗First posted 27 Oct 2022cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Advances in self-supervised learning have significantly reduced the amount of transcribed audio required for training. However, the majority of work in this area is focused on read speech. We explore limited supervision in the domain of conversational speech. While we assume the amount of in-domain data is limited, we augment the model with open source read speech data. The XLS-R model has been shown to perform well with limited adaptation data and serves as a strong baseline. We use untranscribed data for self-supervised learning and semi-supervised training in an autoregressive encoder-decoder model. We demonstrate that by using the XLS-R model for pseudotranscription, a much smaller autoregressive model can outperform a finetuned XLS-R model when transcribed in-domain data is limited, reducing WER by as much as 8% absolute.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

26 of 41 references.

References · 26

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  10. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  11. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  12. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
14 more
  1. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  2. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  3. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 88
  4. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  5. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  6. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 45
  7. 2020Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Trainingcited by 39
  8. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27
  9. 2021Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognitioncited by 21
  10. 2020A Crowdsourced Open-Source Kazakh Speech Corpus and Initial Speech Recognition Baselinecited by 14
  11. 2017Comparison of Decoding Strategies for CTC Acoustic Modelscited by 13
  12. 2021Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscriptscited by 3
  13. 2021Injecting Text and Cross-lingual Supervision in Few-shot Learning from Self-Supervised Modelscited by 2
  14. 2021Combining Unsupervised and Text Augmented Semi-Supervised Learning for Low Resourced Autoregressive Speech Recognitioncited by 2

Built on by

Not cited yet.