mlsanityData to 14 Oct 2026

Improving Transformer-based Speech Recognition Using Unsupervised Pre-training

arXiv 1910.09932 ↗First posted 22 Oct 2019cs.CL cs.SD eess.AScited by 30

Not on the map. Why?

Abstract (arXiv)

Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, we can further reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 11; 2023: 6; 2024: 1; 2025: 1.

30 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2021Unsupervised Speech Recognitioncited by 148
  3. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  6. 2021CLSRIL-23: Cross Lingual Speech Representations for Indic Languagescited by 22
  7. 2021Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognitioncited by 17
  8. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  9. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Modelscited by 15
  10. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  11. 2023Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASRcited by 10
  12. 2021CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representationscited by 9
18 more
  1. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  2. 2022Adaptive multilingual speech recognition with pretrained modelscited by 9
  3. 2022PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputationcited by 9
  4. 2021Improving speech recognition models with small samples for air traffic control systemscited by 6
  5. 2021Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASRcited by 5
  6. 2021Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL Systemcited by 4
  7. 2021Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcriptioncited by 3
  8. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  9. 2023Indonesian Automatic Speech Recognition with XLSR-53cited by 2
  10. 2021SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognitioncited by 1
  11. 2022Speech Augmentation Based Unsupervised Learning for Keyword Spottingcited by 1
  12. 2022Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approachcited by 1
  13. 2022Speaker Identification using Speech Recognitioncited by 0
  14. 2022MET: Masked Encoding for Tabular Datacited by 0
  15. 2023Dual Learning for Large Vocabulary On-Device ASRcited by 0
  16. 2023Self-supervised speech representation learning for keyword-spotting with light-weight transformerscited by 0
  17. 2024Contrastive Augmentation: An Unsupervised Learning Approach for Keyword Spotting in Speech Technologycited by 0
  18. 2025Learning Speech Representations with Variational Predictive Codingcited by 0