# Simple Fusion: Return of the Language Model

arXiv: https://arxiv.org/abs/1809.00125 · HTML: https://mlsanity.com/paper/1809.00125/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2018.

## Built on by (most cited first, 30 of 30)

- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Decoding-time Realignment of Language Models](https://mlsanity.com/paper/2402.02992/) (2024)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT](https://mlsanity.com/paper/2109.01396/) (2021)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [Spell my name: keyword boosted speech recognition](https://mlsanity.com/paper/2110.02791/) (2021)
- [Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR](https://mlsanity.com/paper/2201.11627/) (2022)
- [Mukayese: Turkish NLP Strikes Back](https://mlsanity.com/paper/2203.01215/) (2022)
- [Code Generation from Natural Language with Less Prior and More Monolingual Data](https://mlsanity.com/paper/2101.00259/) (2021)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [Is Encoder-Decoder Redundant for Neural Machine Translation?](https://mlsanity.com/paper/2210.11807/) (2022)
- [Sequentially Controlled Text Generation](https://mlsanity.com/paper/2301.02299/) (2023)
- [On-the-Fly Fusion of Large Language Models and Machine Translation](https://mlsanity.com/paper/2311.08306/) (2023)
- [Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition](https://mlsanity.com/paper/2501.09258/) (2025)
- [Controlled Cue Generation for Play Scripts](https://mlsanity.com/paper/2112.06953/) (2021)
- [Twist Decoding: Diverse Generators Guide Each Other](https://mlsanity.com/paper/2205.09273/) (2022)
- [Reference-less Analysis of Context Specificity in Translation with Personalised Language Models](https://mlsanity.com/paper/2303.16618/) (2023)
- [CharED: Character-wise Ensemble Decoding for Large Language Models](https://mlsanity.com/paper/2407.11009/) (2024)
- [Improving Language Model Integration for Neural Machine Translation](https://mlsanity.com/paper/2306.05077/) (2023)
- [Deep CLAS: Deep Contextual Listen, Attend and Spell](https://mlsanity.com/paper/2409.17603/) (2024)
- [Infusing Future Information into Monotonic Attention Through Language Models](https://mlsanity.com/paper/2109.03121/) (2021)
- [Interplay of Machine Translation, Diacritics, and Diacritization](https://mlsanity.com/paper/2404.05943/) (2024)
- [Q-FAKER: Query-free Hard Black-box Attack via Controlled Generation](https://mlsanity.com/paper/2504.13551/) (2025)
- [Topical Language Generation using Transformers](https://mlsanity.com/paper/2103.06434/) (2021)
- [Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation](https://mlsanity.com/paper/2212.12662/) (2022)
- [Plug, Play, and Fuse: Zero-Shot Joint Decoding via Word-Level Re-ranking Across Diverse Vocabularies](https://mlsanity.com/paper/2408.11327/) (2024)

## Abstract (arXiv)

Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for NMT training: We combine the scores of a pre-trained and fixed language model (LM) with the scores of a translation model (TM) while the TM is trained from scratch. To achieve that, we train the translation model to predict the residual probability of the training data added to the prediction of the LM. This enables the TM to focus its capacity on modeling the source sentence since it can rely on the LM for fluency. We show that our method outperforms previous approaches to integrate LMs into NMT while the architecture is simpler as it does not require gating networks to balance TM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test sets (English-Turkish, Turkish-English, Estonian-English, Xhosa-English) on top of ensembles without LM. We compare our method with alternative ways to utilize monolingual data such as backtranslation, shallow fusion, and cold fusion.