# Semi-Supervised Speech Recognition via Local Prior Matching

arXiv: https://arxiv.org/abs/2002.10336 · HTML: https://mlsanity.com/paper/2002.10336/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 13 of 13)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition](https://mlsanity.com/paper/2106.07759/) (2021)
- [EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition](https://mlsanity.com/paper/2104.07474/) (2021)
- [Continuous Soft Pseudo-Labeling in ASR](https://mlsanity.com/paper/2211.06007/) (2022)
- [A Unified Speaker Adaptation Approach for ASR](https://mlsanity.com/paper/2110.08545/) (2021)
- [Filter and evolve: progressive pseudo label refining for semi-supervised automatic speech recognition](https://mlsanity.com/paper/2210.16318/) (2022)
- [Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation](https://mlsanity.com/paper/2108.01812/) (2021)
- [InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss](https://mlsanity.com/paper/2211.00795/) (2022)
- [Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information](https://mlsanity.com/paper/2212.03476/) (2022)

## Abstract (arXiv)

For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability. In this work, we propose local prior matching (LPM), a semi-supervised objective that distills knowledge from a strong prior (e.g. a language model) to provide learning signal to a discriminative model trained on unlabeled speech. We demonstrate that LPM is theoretically well-motivated, simple to implement, and superior to existing knowledge distillation techniques under comparable settings. Starting from a baseline trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data, LPM recovers 54% and 73% of the word error rate on clean and noisy test sets relative to a fully supervised model on the same data.