# Acoustic Model Fusion for End-to-end Speech Recognition

arXiv: https://arxiv.org/abs/2310.07062 · HTML: https://mlsanity.com/paper/2310.07062/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition](https://mlsanity.com/paper/1910.01992/) (2019)

## Built on by (most cited first, 6 of 6)

- [Contextualization of ASR with LLM using phonetic retrieval-based augmentation](https://mlsanity.com/paper/2409.15353/) (2024)
- [Personalization of CTC-based End-to-End Speech Recognition Using Pronunciation-Driven Subword Tokenization](https://mlsanity.com/paper/2310.09988/) (2023)
- [Transformer-based Model for ASR N-Best Rescoring and Rewriting](https://mlsanity.com/paper/2406.08207/) (2024)
- [Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition](https://mlsanity.com/paper/2510.19471/) (2025)
- [Enhancing CTC-based speech recognition with diverse modeling units](https://mlsanity.com/paper/2406.03274/) (2024)
- [Phonetically-Augmented Discriminative Rescoring for Voice Search Error Correction](https://mlsanity.com/paper/2506.06117/) (2025)

## Abstract (arXiv)

Recent advances in deep learning and automatic speech recognition (ASR) have enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new level. The E2E systems implicitly model all conventional ASR components, such as the acoustic model (AM) and the language model (LM), in a single network trained on audio-text pairs. Despite this simpler system architecture, fusing a separate LM, trained exclusively on text corpora, into the E2E system has proven to be beneficial. However, the application of LM fusion presents certain drawbacks, such as its inability to address the domain mismatch issue inherent to the internal AM. Drawing inspiration from the concept of LM fusion, we propose the integration of an external AM into the E2E system to better address the domain mismatch. By implementing this novel approach, we have achieved a significant reduction in the word error rate, with an impressive drop of up to 14.3% across varied test sets. We also discovered that this AM fusion approach is particularly beneficial in enhancing named entity recognition.