# End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures

arXiv: https://arxiv.org/abs/1911.08460 · HTML: https://mlsanity.com/paper/1911.08460/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2019.

## Built on by (most cited first, 30 of 120)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Federated Learning with Partial Model Personalization](https://mlsanity.com/paper/2204.03809/) (2022)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers](https://mlsanity.com/paper/2301.04545/) (2023)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Towards Building ASR Systems for the Next Billion Users](https://mlsanity.com/paper/2111.03945/) (2021)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [3D Siamese Transformer Network for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2207.11995/) (2022)
- [SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training](https://mlsanity.com/paper/2210.03730/) (2022)
- [An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04590/) (2021)
- [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://mlsanity.com/paper/2104.06678/) (2021)
- [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021)

## Abstract (arXiv)

We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.