# Early Stage LM Integration Using Local and Global Log-Linear Combination

arXiv: https://arxiv.org/abs/2005.10049 · HTML: https://mlsanity.com/paper/2005.10049/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 9 of 9)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model](https://mlsanity.com/paper/2201.01995/) (2022)
- [Efficient Sequence Training of Attention Models using Approximative Recombination](https://mlsanity.com/paper/2110.09245/) (2021)
- [Integrating Lattice-Free MMI into End-to-End Speech Recognition](https://mlsanity.com/paper/2203.15614/) (2022)
- [On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers](https://mlsanity.com/paper/2309.14130/) (2023)
- [Automatic Learning of Subword Dependent Model Scales](https://mlsanity.com/paper/2110.09324/) (2021)

## Abstract (arXiv)

Sequence-to-sequence models with an implicit alignment mechanism (e.g. attention) are closing the performance gap towards traditional hybrid hidden Markov models (HMM) for the task of automatic speech recognition. One important factor to improve word error rate in both cases is the use of an external language model (LM) trained on large text-only corpora. Language model integration is straightforward with the clear separation of acoustic model and language model in classical HMM-based modeling. In contrast, multiple integration schemes have been proposed for attention models. In this work, we present a novel method for language model integration into implicit-alignment based sequence-to-sequence models. Log-linear model combination of acoustic and language model is performed with a per-token renormalization. This allows us to compute the full normalization term efficiently both in training and in testing. This is compared to a global renormalization scheme which is equivalent to applying shallow fusion in training. The proposed methods show good improvements over standard model combination (shallow fusion) on our state-of-the-art Librispeech system. Furthermore, the improvements are persistent even if the LM is exchanged for a more powerful one after training.