# An Empirical Study of In-context Learning in LLMs for Machine Translation

arXiv: https://arxiv.org/abs/2401.12097 · HTML: https://mlsanity.com/paper/2401.12097/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Transformers as Algorithms: Generalization and Stability in In-context Learning](https://mlsanity.com/paper/2301.07067/) (2023)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance](https://mlsanity.com/paper/2401.03729/) (2024)
- [Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples](https://mlsanity.com/paper/1803.01128/) (2018)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)

## Built on by (most cited first, 9 of 9)

- [Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation](https://mlsanity.com/paper/2503.06594/) (2025)
- [Calibrating Uncertainty Quantification of Multi-Modal LLMs using Grounding](https://mlsanity.com/paper/2505.03788/) (2025)
- [Pruning Multilingual Large Language Models for Multilingual Inference](https://mlsanity.com/paper/2409.16911/) (2024)
- [Compensating for Data with Reasoning: Low-Resource Machine Translation with LLMs](https://mlsanity.com/paper/2505.22293/) (2025)
- [Benchmarking Machine Translation on Chinese Social Media Texts](https://mlsanity.com/paper/2601.22931/) (2026)
- [Beyond Many-Shot Translation: Scaling In-Context Demonstrations For Low-Resource Machine Translation](https://mlsanity.com/paper/2602.04764/) (2026)
- [When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking](https://mlsanity.com/paper/2606.31087/) (2026)
- [Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models](https://mlsanity.com/paper/2507.03726/) (2025)
- [Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?](https://mlsanity.com/paper/2603.16660/) (2026)

## Abstract (arXiv)

Recent interest has surged in employing Large Language Models (LLMs) for machine translation (MT) via in-context learning (ICL) (Vilar et al., 2023). Most prior studies primarily focus on optimizing translation quality, with limited attention to understanding the specific aspects of ICL that influence the said quality. To this end, we perform the first of its kind, an exhaustive study of in-context learning for machine translation. We first establish that ICL is primarily example-driven and not instruction-driven. Following this, we conduct an extensive exploration of various aspects of the examples to understand their influence on downstream performance. Our analysis includes factors such as quality and quantity of demonstrations, spatial proximity, and source versus target originality. Further, we also investigate challenging scenarios involving indirectness and misalignment of examples to understand the limits of ICL. While we establish the significance of the quality of the target distribution over the source distribution of demonstrations, we further observe that perturbations sometimes act as regularizers, resulting in performance improvements. Surprisingly, ICL does not necessitate examples from the same task, and a related task with the same target distribution proves sufficient. We hope that our study acts as a guiding resource for considerations in utilizing ICL for MT. Our code is available on https://github.com/PranjalChitale/in-context-mt-analysis.