# A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition

arXiv: https://arxiv.org/abs/1807.10857 · HTML: https://mlsanity.com/paper/1807.10857/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 30 of 66)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizers](https://mlsanity.com/paper/2202.07894/) (2022)
- [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021)
- [Transformer Language Models with LSTM-based Cross-utterance Information Representation](https://mlsanity.com/paper/2102.06474/) (2021)
- [Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study](https://mlsanity.com/paper/2303.06740/) (2023)
- [Efficient Domain Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2302.01496/) (2023)
- [Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation](https://mlsanity.com/paper/2301.13003/) (2023)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation](https://mlsanity.com/paper/2410.13198/) (2024)
- [Lookup-Table Recurrent Language Models for Long Tail Speech Recognition](https://mlsanity.com/paper/2104.04552/) (2021)
- [Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR](https://mlsanity.com/paper/2201.11627/) (2022)
- [Language model fusion for streaming end to end speech recognition](https://mlsanity.com/paper/2104.04487/) (2021)
- [Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognition](https://mlsanity.com/paper/2110.04891/) (2021)
- [Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model](https://mlsanity.com/paper/2201.01995/) (2022)
- [Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2210.15398/) (2022)
- [Acoustic Model Fusion for End-to-end Speech Recognition](https://mlsanity.com/paper/2310.07062/) (2023)
- [On-Device Personalization of Automatic Speech Recognition Models for Disordered Speech](https://mlsanity.com/paper/2106.10259/) (2021)
- [Joint Audio/Text Training for Transformer Rescorer of Streaming Speech Recognition](https://mlsanity.com/paper/2211.00174/) (2022)
- [Adding guardrails to advanced chatbots](https://mlsanity.com/paper/2306.07500/) (2023)
- [Revisiting ASR Error Correction with Specialized Models](https://mlsanity.com/paper/2405.15216/) (2024)
- [Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition](https://mlsanity.com/paper/2501.09258/) (2025)
- [GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems](https://mlsanity.com/paper/2501.10734/) (2025)
- [Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks](https://mlsanity.com/paper/2103.13329/) (2021)

## Abstract (arXiv)

Attention-based recurrent neural encoder-decoder models present an elegant solution to the automatic speech recognition problem. This approach folds the acoustic model, pronunciation model, and language model into a single network and requires only a parallel corpus of speech and text for training. However, unlike in conventional approaches that combine separate acoustic and language models, it is not clear how to use additional (unpaired) text. While there has been previous work on methods addressing this problem, a thorough comparison among methods is still lacking. In this paper, we compare a suite of past methods and some of our own proposed methods for using unpaired text data to improve encoder-decoder models. For evaluation, we use the medium-sized Switchboard data set and the large-scale Google voice search and dictation data sets. Our results confirm the benefits of using unpaired text across a range of methods and data sets. Surprisingly, for first-pass decoding, the rather simple approach of shallow fusion performs best across data sets. However, for Google data sets we find that cold fusion has a lower oracle error rate and outperforms other approaches after second-pass rescoring on the Google voice search data set.