# Improving Transformer-based Speech Recognition Using Unsupervised Pre-training

arXiv: https://arxiv.org/abs/1910.09932 · HTML: https://mlsanity.com/paper/1910.09932/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2019.

## Built on by (most cited first, 30 of 30)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2210.08634/) (2022)
- [CLSRIL-23: Cross Lingual Speech Representations for Indic Languages](https://mlsanity.com/paper/2107.07402/) (2021)
- [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021)
- [MelHuBERT: A simplified HuBERT on Mel spectrograms](https://mlsanity.com/paper/2211.09944/) (2022)
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers](https://mlsanity.com/paper/2310.09925/) (2023)
- [Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR](https://mlsanity.com/paper/2304.04974/) (2023)
- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)
- [PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputation](https://mlsanity.com/paper/2212.07514/) (2022)
- [Improving speech recognition models with small samples for air traffic control systems](https://mlsanity.com/paper/2102.08015/) (2021)
- [Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR](https://mlsanity.com/paper/2102.06816/) (2021)
- [Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL System](https://mlsanity.com/paper/2106.09963/) (2021)
- [Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcription](https://mlsanity.com/paper/2112.08692/) (2021)
- [On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation](https://mlsanity.com/paper/2307.02720/) (2023)
- [Indonesian Automatic Speech Recognition with XLSR-53](https://mlsanity.com/paper/2308.11589/) (2023)
- [SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04187/) (2021)
- [Speech Augmentation Based Unsupervised Learning for Keyword Spotting](https://mlsanity.com/paper/2205.14329/) (2022)
- [Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approach](https://mlsanity.com/paper/2210.13805/) (2022)
- [Speaker Identification using Speech Recognition](https://mlsanity.com/paper/2205.14649/) (2022)
- [MET: Masked Encoding for Tabular Data](https://mlsanity.com/paper/2206.08564/) (2022)
- [Dual Learning for Large Vocabulary On-Device ASR](https://mlsanity.com/paper/2301.04327/) (2023)
- [Self-supervised speech representation learning for keyword-spotting with light-weight transformers](https://mlsanity.com/paper/2303.04255/) (2023)
- [Contrastive Augmentation: An Unsupervised Learning Approach for Keyword Spotting in Speech Technology](https://mlsanity.com/paper/2409.00356/) (2024)
- [Learning Speech Representations with Variational Predictive Coding](https://mlsanity.com/paper/2601.00100/) (2025)

## Abstract (arXiv)

Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, we can further reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.