# Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2104.09106 · HTML: https://mlsanity.com/paper/2104.09106/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020)
- [Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions](https://mlsanity.com/paper/1904.02619/) (2019)
- [From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.01493/) (2019)
- [Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/2010.16368/) (2020)
- [The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment](https://mlsanity.com/paper/2004.00960/) (2020)
- [A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition](https://mlsanity.com/paper/1606.06871/) (2016)
- [Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling](https://mlsanity.com/paper/1811.04284/) (2018)
- [Latent Sequence Decompositions](https://mlsanity.com/paper/1610.03035/) (2016)
- [Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling](https://mlsanity.com/paper/1703.00096/) (2017)
- [Subword Regularization: An Analysis of Scalability and Generalization for End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2008.04034/) (2020)

## Built on by (most cited first, 11 of 11)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Units](https://mlsanity.com/paper/2110.04109/) (2021)
- [Exploring the Benefits of Tokenization of Discrete Acoustic Units](https://mlsanity.com/paper/2406.05547/) (2024)
- [RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition](https://mlsanity.com/paper/2305.17782/) (2023)
- [On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers](https://mlsanity.com/paper/2309.14130/) (2023)
- [Syllable Subword Tokens for Open Vocabulary Speech Recognition in Malayalam](https://mlsanity.com/paper/2301.06736/) (2023)
- [Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition](https://mlsanity.com/paper/2401.15532/) (2024)

## Abstract (arXiv)

Subword units are commonly used for end-to-end automatic speech recognition (ASR), while a fully acoustic-oriented subword modeling approach is somewhat missing. We propose an acoustic data-driven subword modeling (ADSM) approach that adapts the advantages of several text-based and acoustic-based subword methods into one pipeline. With a fully acoustic-oriented label design and learning process, ADSM produces acoustic-structured subword units and acoustic-matched target sequence for further ASR training. The obtained ADSM labels are evaluated with different end-to-end ASR approaches including CTC, RNN-Transducer and attention models. Experiments on the LibriSpeech corpus show that ADSM clearly outperforms both byte pair encoding (BPE) and pronunciation-assisted subword modeling (PASM) in all cases. Detailed analysis shows that ADSM achieves acoustically more logical word segmentation and more balanced sequence length, and thus, is suitable for both time-synchronous and label-synchronous models. We also briefly describe how to apply acoustic-based subword regularization and unseen text segmentation using ADSM.