# Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2309.15796 · HTML: https://mlsanity.com/paper/2309.15796/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching](https://mlsanity.com/paper/1812.09323/) (2018)
- [CTC Variations Through New WFST Topologies](https://mlsanity.com/paper/2110.03098/) (2021)
- [Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications](https://mlsanity.com/paper/2010.14665/) (2020)
- [Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR](https://mlsanity.com/paper/2111.06799/) (2021)
- [Star Temporal Classification: Sequence Classification with Partially Labeled Data](https://mlsanity.com/paper/2201.12208/) (2022)
- [Bypass Temporal Classification: Weakly Supervised Automatic Speech Recognition with Imperfect Transcripts](https://mlsanity.com/paper/2306.01031/) (2023)
- [Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2308.06547/) (2023)
- [EURO: ESPnet Unsupervised ASR Open-source Toolkit](https://mlsanity.com/paper/2211.17196/) (2022)
- [Lattice-based lightly-supervised acoustic model training](https://mlsanity.com/paper/1905.13150/) (2019)
- [HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation](https://mlsanity.com/paper/2306.11252/) (2023)

## Built on by (most cited first, 8 of 8)

- [Token-Weighted RNN-T for Learning from Flawed Data](https://mlsanity.com/paper/2406.18108/) (2024)
- [Advancing Arabic Speech Recognition Through Large-Scale Weakly Supervised Learning](https://mlsanity.com/paper/2504.12254/) (2025)
- [RNN-Transducer-based Losses for Speech Recognition on Noisy Targets](https://mlsanity.com/paper/2504.06963/) (2025)
- [WST: Weakly Supervised Transducer for Automatic Speech Recognition](https://mlsanity.com/paper/2511.04035/) (2025)
- [Graph Connectionist Temporal Classification for Phoneme Recognition](https://mlsanity.com/paper/2509.05399/) (2025)
- [Modeling Overlapped Speech with Shuffles](https://mlsanity.com/paper/2603.17769/) (2026)
- [Constrained CTC Decoding for Efficient Diacritic Restoration](https://mlsanity.com/paper/2607.18946/) (2026)
- [A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition](https://mlsanity.com/paper/2609.30160/) (2026)

## Abstract (arXiv)

Training automatic speech recognition (ASR) systems requires large amounts of well-curated paired data. However, human annotators usually perform "non-verbatim" transcription, which can result in poorly trained models. In this paper, we propose Omni-temporal Classification (OTC), a novel training criterion that explicitly incorporates label uncertainties originating from such weak supervision. This allows the model to effectively learn speech-text alignments while accommodating errors present in the training transcripts. OTC extends the conventional CTC objective for imperfect transcripts by leveraging weighted finite state transducers. Through experiments conducted on the LibriSpeech and LibriVox datasets, we demonstrate that training ASR models with OTC avoids performance degradation even with transcripts containing up to 70% errors, a scenario where CTC models fail completely. Our implementation is available at https://github.com/k2-fsa/icefall.