mlsanityData to 7 Oct 2026

Language Models are Good Translators

arXiv 2106.13627 ↗First posted 25 Jun 2021cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the rapid advance in neural machine translation (NMT), the core of which lies in the encoder-decoder architecture. Inspired by the recent progress of large-scale pre-trained language models on machine translation in a limited scenario, we firstly demonstrate that a single language model (LM4MT) can achieve comparable performance with strong encoder-decoder NMT models on standard machine translation benchmarks, using the same training data and similar amount of model parameters. LM4MT can also easily utilize source-side texts as additional supervision. Though modeling the source- and target-language texts with the same mechanism, LM4MT can provide unified representations for both source and target sentences, which can better transfer knowledge across languages. Extensive experiments on pivot-based and zero-shot translation tasks show that LM4MT can outperform the encoder-decoder NMT model by a large margin.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
16 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  3. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  4. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  5. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  6. 2018Scaling Neural Machine Translationcited by 204
  7. 2019The Evolved Transformercited by 180
  8. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  9. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  10. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  11. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  12. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  13. 2019Translationese in Machine Translation Evaluationcited by 38
  14. 2019Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languagescited by 36
  15. 2021On the Language Coverage Bias for Neural Machine Translationcited by 12
  16. 2018Generative Neural Machine Translationcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 5; 2023: 4; 2024: 5.

16 papers.

Sort:

By area:

  1. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  2. 2023Auditing large language models: a three-layered approachcited by 135
  3. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  4. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  5. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  6. 2022Examining Scaling and Transfer of Language Model Architectures for Machine Translationcited by 26
  7. 2022Domain-Specific Text Generation for Machine Translationcited by 12
  8. 2024Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directionscited by 12
  9. 2024Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Languagecited by 12
  10. 2022Understanding and Mitigating the Uncertainty in Zero-Shot Translationcited by 9
  11. 2023State Spaces Aren't Enough: Machine Translation Needs Attentioncited by 9
  12. 2023Cultural Adaptation of Recipescited by 6
4 more
  1. 2022Is Encoder-Decoder Redundant for Neural Machine Translation?cited by 5
  2. 2024Language Modelling Approaches to Adaptive Machine Translationcited by 3
  3. 2024A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Contentcited by 1
  4. 2024Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?cited by 0