# Enhancing CTC-based speech recognition with diverse modeling units

arXiv: https://arxiv.org/abs/2406.03274 · HTML: https://mlsanity.com/paper/2406.03274/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020)
- [Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices](https://mlsanity.com/paper/2207.08988/) (2022)
- [Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Units](https://mlsanity.com/paper/2110.04109/) (2021)
- [Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling](https://mlsanity.com/paper/1811.04284/) (2018)
- [Acoustic Model Fusion for End-to-end Speech Recognition](https://mlsanity.com/paper/2310.07062/) (2023)
- [Personalization of CTC-based End-to-End Speech Recognition Using Pronunciation-Driven Subword Tokenization](https://mlsanity.com/paper/2310.09988/) (2023)
- [Conformer-Based Speech Recognition On Extreme Edge-Computing Devices](https://mlsanity.com/paper/2312.10359/) (2023)
- [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021)
- [Phonetic-assisted Multi-Target Units Modeling for Improving Conformer-Transducer ASR system](https://mlsanity.com/paper/2211.01571/) (2022)

## Built on by (most cited first, 1 of 1)

- [T-SANDHI: Tone Sandhi-aware Adaptive Network with Decoupled Hybrid Injection for Low-resource Taiwanese Hokkien Speech Recognition](https://mlsanity.com/paper/2609.18194/) (2026)

## Abstract (arXiv)

In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.