# Integrating Lattice-Free MMI into End-to-End Speech Recognition arXiv: https://arxiv.org/abs/2203.15614 ยท HTML: https://mlsanity.com/paper/2203.15614/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 58 references that are in the map. - First posted on arXiv 29 Mar 2022. ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018) - [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015) - [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021) - [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017) - [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020) - [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020) - [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018) - [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021) - [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020) - [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017) - [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017) - [Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition](https://mlsanity.com/paper/2007.13802/) (2020) - [Deep-FSMN for Large Vocabulary Continuous Speech Recognition](https://mlsanity.com/paper/1803.05030/) (2018) - [Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition](https://mlsanity.com/paper/1911.12487/) (2019) - [Differentiable Weighted Finite-State Transducers](https://mlsanity.com/paper/2010.01003/) (2020) - [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021) - [PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR](https://mlsanity.com/paper/2005.09824/) (2020) - [The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans](https://mlsanity.com/paper/2012.13006/) (2020) - [On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2010.12673/) (2020) - [Early Stage LM Integration Using Local and Global Log-Linear Combination](https://mlsanity.com/paper/2005.10049/) (2020) - [Consistent Training and Decoding For End-to-end Speech Recognition Using Lattice-free MMI](https://mlsanity.com/paper/2112.02498/) (2021) - [On lattice-free boosted MMI training of HMM and CTC-based full-context ASR models](https://mlsanity.com/paper/2107.04154/) (2021) - [Improving Proper Noun Recognition in End-to-End ASR By Customization of the MWER Loss Criterion](https://mlsanity.com/paper/2005.09756/) (2020) - [Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model](https://mlsanity.com/paper/2201.01995/) (2022) ## Built on by (most cited first, 3 of 3) - [Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation](https://mlsanity.com/paper/2301.13003/) (2023) - [Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers](https://mlsanity.com/paper/2212.04325/) (2022) - [Investigating the Effect of Language Models in Sequence Discriminative Training for Neural Transducers](https://mlsanity.com/paper/2310.07345/) (2023) ## Abstract (arXiv) In automatic speech recognition (ASR) research, discriminative criteria have achieved superior performance in DNN-HMM systems. Given this success, the adoption of discriminative criteria is promising to boost the performance of end-to-end (E2E) ASR systems. With this motivation, previous works have introduced the minimum Bayesian risk (MBR, one of the discriminative criteria) into E2E ASR systems. However, the effectiveness and efficiency of the MBR-based methods are compromised: the MBR criterion is only used in system training, which creates a mismatch between training and decoding; the on-the-fly decoding process in MBR-based methods results in the need for pre-trained models and slow training speeds. To this end, novel algorithms are proposed in this work to integrate another widely used discriminative criterion, lattice-free maximum mutual information (LF-MMI), into E2E ASR systems not only in the training stage but also in the decoding process. The proposed LF-MMI training and decoding methods show their effectiveness on two widely used E2E frameworks: Attention-Based Encoder-Decoders (AEDs) and Neural Transducers (NTs). Compared with MBR-based methods, the proposed LF-MMI method: maintains the consistency between training and decoding; eschews the on-the-fly decoding process; trains from randomly initialized models with superior training efficiency. Experiments suggest that the LF-MMI method outperforms its MBR counterparts and consistently leads to statistically significant performance improvements on various frameworks and datasets from 30 hours to 14.3k hours. The proposed method achieves state-of-the-art (SOTA) results on Aishell-1 (CER 4.10%) and Aishell-2 (CER 5.02%) datasets. Code is released.