# Language Model Prior for Low-Resource Neural Machine Translation

arXiv: https://arxiv.org/abs/2004.14928 · HTML: https://mlsanity.com/paper/2004.14928/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 37)

- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [BERTTune: Fine-Tuning Neural Machine Translation with BERTScore](https://mlsanity.com/paper/2106.02208/) (2021)
- [Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling](https://mlsanity.com/paper/2102.03551/) (2021)
- [IndT5: A Text-to-Text Transformer for 10 Indigenous Languages](https://mlsanity.com/paper/2104.07483/) (2021)
- [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023)
- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2303.15265/) (2023)
- [Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation](https://mlsanity.com/paper/2202.13663/) (2022)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [A Theory of Unsupervised Translation Motivated by Understanding Animal Communication](https://mlsanity.com/paper/2211.11081/) (2022)
- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [The Role of n-gram Smoothing in the Age of Neural Networks](https://mlsanity.com/paper/2403.17240/) (2024)
- [TopicRefine: Joint Topic Prediction and Dialogue Response Generation for Multi-turn End-to-End Dialogue System](https://mlsanity.com/paper/2109.05187/) (2021)
- [Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models](https://mlsanity.com/paper/2409.17836/) (2024)
- [Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation](https://mlsanity.com/paper/2104.05848/) (2021)
- [Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation](https://mlsanity.com/paper/2107.13077/) (2021)
- [Learning to translate by learning to communicate](https://mlsanity.com/paper/2207.07025/) (2022)
- [ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2212.04262/) (2022)
- [BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation](https://mlsanity.com/paper/2111.06787/) (2021)
- [How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction](https://mlsanity.com/paper/2111.03108/) (2021)
- [Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation](https://mlsanity.com/paper/2412.19522/) (2024)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Improving Similar Language Translation With Transfer Learning](https://mlsanity.com/paper/2108.03533/) (2021)

## Abstract (arXiv)

The scarcity of large parallel corpora is an important obstacle for neural machine translation. A common solution is to exploit the knowledge of language models (LM) trained on abundant monolingual data. In this work, we propose a novel approach to incorporate a LM as prior in a neural translation model (TM). Specifically, we add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior, while avoiding wrong predictions when the TM "disagrees" with the LM. This objective relates to knowledge distillation, where the LM can be viewed as teaching the TM about the target language. The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previous work that requires it during inference. We present an analysis of the effects that different methods have on the distributions of the TM. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data.