# Language Models are Good Translators

arXiv: https://arxiv.org/abs/2106.13627 · HTML: https://mlsanity.com/paper/2106.13627/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://mlsanity.com/paper/1906.01181/) (2019)
- [On the Inference Calibration of Neural Machine Translation](https://mlsanity.com/paper/2005.00963/) (2020)
- [The Effect of Translationese in Machine Translation Test Sets](https://mlsanity.com/paper/1906.08069/) (2019)
- [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019)
- [Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages](https://mlsanity.com/paper/1909.09524/) (2019)
- [On the Language Coverage Bias for Neural Machine Translation](https://mlsanity.com/paper/2106.03297/) (2021)
- [Generative Neural Machine Translation](https://mlsanity.com/paper/1806.05138/) (2018)

## Built on by (most cited first, 16 of 16)

- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Adaptive Machine Translation with Large Language Models](https://mlsanity.com/paper/2301.13294/) (2023)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators](https://mlsanity.com/paper/2110.06609/) (2021)
- [Examining Scaling and Transfer of Language Model Architectures for Machine Translation](https://mlsanity.com/paper/2202.00528/) (2022)
- [Domain-Specific Text Generation for Machine Translation](https://mlsanity.com/paper/2208.05909/) (2022)
- [Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directions](https://mlsanity.com/paper/2401.08429/) (2024)
- [Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language](https://mlsanity.com/paper/2404.04809/) (2024)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [State Spaces Aren't Enough: Machine Translation Needs Attention](https://mlsanity.com/paper/2304.12776/) (2023)
- [Cultural Adaptation of Recipes](https://mlsanity.com/paper/2310.17353/) (2023)
- [Is Encoder-Decoder Redundant for Neural Machine Translation?](https://mlsanity.com/paper/2210.11807/) (2022)
- [Language Modelling Approaches to Adaptive Machine Translation](https://mlsanity.com/paper/2401.14559/) (2024)
- [A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content](https://mlsanity.com/paper/2410.03277/) (2024)
- [Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?](https://mlsanity.com/paper/2410.06338/) (2024)

## Abstract (arXiv)

Recent years have witnessed the rapid advance in neural machine translation (NMT), the core of which lies in the encoder-decoder architecture. Inspired by the recent progress of large-scale pre-trained language models on machine translation in a limited scenario, we firstly demonstrate that a single language model (LM4MT) can achieve comparable performance with strong encoder-decoder NMT models on standard machine translation benchmarks, using the same training data and similar amount of model parameters. LM4MT can also easily utilize source-side texts as additional supervision. Though modeling the source- and target-language texts with the same mechanism, LM4MT can provide unified representations for both source and target sentences, which can better transfer knowledge across languages. Extensive experiments on pivot-based and zero-shot translation tasks show that LM4MT can outperform the encoder-decoder NMT model by a large margin.