# A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2609.30160 · HTML: https://mlsanity.com/paper/2609.30160/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Differentiable Weighted Finite-State Transducers](https://mlsanity.com/paper/2010.01003/) (2020)
- [Semi-Supervised Speech Recognition via Graph-based Temporal Classification](https://mlsanity.com/paper/2010.15653/) (2020)
- [Bypass Temporal Classification: Weakly Supervised Automatic Speech Recognition with Imperfect Transcripts](https://mlsanity.com/paper/2306.01031/) (2023)
- [Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2309.15796/) (2023)
- [Powerful and Extensible WFST Framework for RNN-Transducer Losses](https://mlsanity.com/paper/2303.10384/) (2023)
- [Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2308.06547/) (2023)
- [Style-agnostic evaluation of ASR using multiple reference transcripts](https://mlsanity.com/paper/2412.07937/) (2024)
- [Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages](https://mlsanity.com/paper/2603.00941/) (2026)
- [NTC-KWS: Noise-aware CTC for Robust Keyword Spotting](https://mlsanity.com/paper/2412.12614/) (2024)
- [Token-Weighted RNN-T for Learning from Flawed Data](https://mlsanity.com/paper/2406.18108/) (2024)
- [WST: Weakly Supervised Transducer for Automatic Speech Recognition](https://mlsanity.com/paper/2511.04035/) (2025)

## Abstract (arXiv)

Automatic speech recognition is typically trained assuming that the reference transcript is the only valid labeling of an utterance, yet even nominally verbatim transcripts contain localized differences in pronunciation, spelling, or lexical realization that the acoustics do not uniquely determine. Omni-temporal Classification (OTC) tolerates such noise by adding wildcard paths to the connectionist temporal classification (CTC) alignment graph, but its word-level arcs are too coarse, since bypassing one unsupported token discards supervision for the whole word. We move wildcard arcs to token granularity so unsupported tokens can be bypassed while the rest of the word stays supervised, and we combine token- and word-level arcs as complementary escape paths. Across 19 languages and three corpora, token-level OTC improves over CTC on all 25 tasks. We also replace epoch-indexed relaxation of the wildcard weights with a predictive-entropy-indexed schedule, which performs comparably while reducing dependence on training length. Combining this schedule with the hybrid graph gives the lowest mean word error rate (WER) on every corpus and a 9.45% average relative WER reduction over CTC. Independent validator transcriptions show that token-level models place significantly more wildcard-bypass probability than CTC on disputed characters, indicating that token-level tolerance targets localized transcript ambiguity.