# Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2308.06547 · HTML: https://mlsanity.com/paper/2308.06547/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](https://mlsanity.com/paper/2205.07246/) (2022)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2003.03773/) (2020)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [SlimIPL: Language-Model-Free Iterative Pseudo-Labeling](https://mlsanity.com/paper/2010.11524/) (2020)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training](https://mlsanity.com/paper/2011.13439/) (2020)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [Semi-supervised ASR by End-to-end Self-training](https://mlsanity.com/paper/2001.09128/) (2020)
- [Differentiable Weighted Finite-State Transducers](https://mlsanity.com/paper/2010.01003/) (2020)
- [Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition](https://mlsanity.com/paper/2106.07759/) (2021)
- [Semi-Supervised Speech Recognition via Graph-based Temporal Classification](https://mlsanity.com/paper/2010.15653/) (2020)
- [CTC Variations Through New WFST Topologies](https://mlsanity.com/paper/2110.03098/) (2021)
- [Incremental Learning for End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2005.04288/) (2020)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [Lead2Gold: Towards exploiting the full potential of noisy transcriptions for speech recognition](https://mlsanity.com/paper/1910.07323/) (2019)
- [Multiple-hypothesis CTC-based semi-supervised adaptation of end-to-end speech recognition](https://mlsanity.com/paper/2103.15515/) (2021)

## Built on by (most cited first, 6 of 6)

- [Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2309.15796/) (2023)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [Token-Weighted RNN-T for Learning from Flawed Data](https://mlsanity.com/paper/2406.18108/) (2024)
- [LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data](https://mlsanity.com/paper/2506.04586/) (2025)
- [Graph Connectionist Temporal Classification for Phoneme Recognition](https://mlsanity.com/paper/2509.05399/) (2025)
- [A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition](https://mlsanity.com/paper/2609.30160/) (2026)

## Abstract (arXiv)

When labeled data is insufficient, semi-supervised learning with the pseudo-labeling technique can significantly improve the performance of automatic speech recognition. However, pseudo-labels are often noisy, containing numerous incorrect tokens. Taking noisy labels as ground-truth in the loss function results in suboptimal performance. Previous works attempted to mitigate this issue by either filtering out the nosiest pseudo-labels or improving the overall quality of pseudo-labels. While these methods are effective to some extent, it is unrealistic to entirely eliminate incorrect tokens in pseudo-labels. In this work, we propose a novel framework named alternative pseudo-labeling to tackle the issue of noisy pseudo-labels from the perspective of the training objective. The framework comprises several components. Firstly, a generalized CTC loss function is introduced to handle noisy pseudo-labels by accepting alternative tokens in the positions of incorrect tokens. Applying this loss function in pseudo-labeling requires detecting incorrect tokens in the predicted pseudo-labels. In this work, we adopt a confidence-based error detection method that identifies the incorrect tokens by comparing their confidence scores with a given threshold, thus necessitating the confidence score to be discriminative. Hence, the second proposed technique is the contrastive CTC loss function that widens the confidence gap between the correctly and incorrectly predicted tokens, thereby improving the error detection ability. Additionally, obtaining satisfactory performance with confidence-based error detection typically requires extensive threshold tuning. Instead, we propose an automatic thresholding method that uses labeled data as a proxy for determining the threshold, thus saving the pain of manual tuning.