# A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale

arXiv: https://arxiv.org/abs/2304.11053 · HTML: https://mlsanity.com/paper/2304.11053/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training](https://mlsanity.com/paper/2110.10329/) (2021)
- [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Cycle-consistency training for end-to-end speech recognition](https://mlsanity.com/paper/1811.01690/) (2018)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition](https://mlsanity.com/paper/2005.10469/) (2020)
- [USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder](https://mlsanity.com/paper/2202.06045/) (2022)

## Built on by (most cited first, 1 of 1)

- [Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition](https://mlsanity.com/paper/2310.13015/) (2023)

## Abstract (arXiv)

Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.