mlsanityData to 7 Oct 2026

The unreasonable effectiveness of few-shot learning for machine translation

arXiv 2302.01398 ↗First posted 2 Feb 2023cs.CLcited by 91

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We demonstrate the potential of few-shot translation systems, trained with unpaired language data, for both high and low-resource language pairs. We show that with only 5 examples of high-quality translation data shown at inference, a transformer decoder-only model trained solely with self-supervised learning, is able to match specialized supervised state-of-the-art models as well as more general commercial translation systems. In particular, we outperform the best performing system on the WMT'21 English - Chinese news translation task by only using five examples of English - Chinese parallel data at inference. Moreover, our approach in building these models does not necessitate joint multilingual training or back-translation, is conceptually simple and shows the potential to extend to the multilingual setting. Furthermore, the resulting models are two orders of magnitude smaller than state-of-the-art language models. We then analyze the factors which impact the performance of few-shot translation systems, and highlight that the quality of the few-shot demonstrations heavily determines the quality of the translations generated by our models. Finally, we show that the few-shot paradigm also provides a way to control certain attributes of the translation -- we show that we are able to control for regional varieties and formality using only a five examples at inference, paving the way towards controllable machine translation systems.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 44 of its 59 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

44 of 59 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
32 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  2. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  3. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  4. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  5. 2021Scaling Laws for Transfercited by 265
  6. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  7. 2022In-context Examples Selection for Machine Translationcited by 172
  8. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  9. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  10. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  11. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  12. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  13. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  14. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  15. 2020When Does Unsupervised Machine Translation Work?cited by 38
  16. 2020When and Why is Unsupervised Neural Machine Translation Useless?cited by 34
  17. 2018Multi-Task Neural Models for Translating Between Styles Within and Across Languagescited by 28
  18. 2021Unsupervised Neural Machine Translation with Generative Language Models Onlycited by 24
  19. 2020A Multilingual View of Unsupervised Machine Translationcited by 23
  20. 2020Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languagescited by 22
  21. 2021Machine Translation into Low-resource Language Varietiescited by 19
  22. 2021Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoningcited by 19
  23. 2021A Few More Examples May Be Worth Billions of Parameterscited by 19
  24. 2022FRMT: A Benchmark for Few-Shot Region-Aware Machine Translationcited by 18
  25. 2021The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21cited by 14
  26. 2022Controlling Translation Formality Using Pre-trained Multilingual Language Modelscited by 14
  27. 2022Finding Memo: Extractive Memorization in Constrained Sequence Generation Taskscited by 12
  28. 2021WeChat Neural Machine Translation Systems for WMT21cited by 11
  29. 2018Neural Machine Translation into Language Varietiescited by 9
  30. 2021NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21cited by 8
  31. 2021Towards Universality in Multilingual Text Rewritingcited by 7
  32. 2021The NiuTrans Machine Translation Systems for WMT21cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 28; 2024: 41; 2025: 15; 2026: 7.

91 papers.

Sort:

By area:

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  3. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  4. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 180
  5. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  6. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  7. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  8. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  9. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  10. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  11. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  12. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
38 more
  1. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  3. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 48
  4. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  5. 2023BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Modelscited by 42
  6. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  7. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  8. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 37
  9. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  10. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  11. 2024Teaching Large Language Models an Unseen Language on the Flycited by 29
  12. 2024G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translationcited by 29
  13. 2023Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decodingcited by 28
  14. 2023Model-as-a-Service (MaaS): A Surveycited by 24
  15. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  16. 2025Seed-X: Building Strong Multilingual Translation LLM with 7B Parameterscited by 21
  17. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  18. 2024How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizescited by 17
  19. 2023An Empirical Study of Translation Hypothesis Ensembling with Large Language Modelscited by 16
  20. 2024DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanismscited by 16
  21. 2024Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimationcited by 13
  22. 2024Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructionscited by 13
  23. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  24. 2024Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directionscited by 12
  25. 2024Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Languagecited by 12
  26. 2024The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMscited by 12
  27. 2024BERTs are Generative In-Context Learnerscited by 11
  28. 2024Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Levelcited by 11
  29. 2024Video Diffusion Transformers are In-Context Learnerscited by 11
  30. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  31. 2023Neural Machine Translation Models Can Learn to be Few-shot Learnerscited by 8
  32. 2023Analyzing Local Representations of Self-supervised Vision Transformerscited by 8
  33. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  34. 2024Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanismcited by 6
  35. 2025Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?cited by 6
  36. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
  37. 2024GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learningcited by 5
  38. 2025Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinementcited by 5

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Do Multilingual Language Models Think Better in English?
  2. 2025FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translation
  3. 2021Revisiting Self-Training for Few-Shot Learning of Language Model
  4. 2025Compensating for Data with Reasoning: Low-Resource Machine Translation with LLMs
  5. 2024Many-Shot In-Context Learning
  6. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  7. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  8. 2025CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation