# Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning

arXiv: https://arxiv.org/abs/2310.13448 · HTML: https://mlsanity.com/paper/2310.13448/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [The unreasonable effectiveness of few-shot learning for machine translation](https://mlsanity.com/paper/2302.01398/) (2023)
- [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020)
- [Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM](https://mlsanity.com/paper/2303.01911/) (2023)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)

## Built on by (most cited first, 30 of 49)

- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [Fine-tuning Large Language Models for Domain-specific Machine Translation](https://mlsanity.com/paper/2402.15061/) (2024)
- [SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain](https://mlsanity.com/paper/2407.19584/) (2024)
- [MaLA-500: Massive Language Adaptation of Large Language Models](https://mlsanity.com/paper/2401.13303/) (2024)
- [Evaluating the Prompt Steerability of Large Language Models](https://mlsanity.com/paper/2411.12405/) (2024)
- [The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities](https://mlsanity.com/paper/2405.20089/) (2024)
- [Fine-tuning Large Language Models for Adaptive Machine Translation](https://mlsanity.com/paper/2312.12740/) (2023)
- [Token-wise Influential Training Data Retrieval for Large Language Models](https://mlsanity.com/paper/2405.11724/) (2024)
- [Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing](https://mlsanity.com/paper/2410.07054/) (2024)
- [Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2401.08350/) (2024)
- [Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?](https://mlsanity.com/paper/2404.14122/) (2024)
- [How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes](https://mlsanity.com/paper/2409.03454/) (2024)
- [Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation](https://mlsanity.com/paper/2406.01806/) (2024)
- [Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?](https://mlsanity.com/paper/2410.24019/) (2024)
- [Understanding In-Context Machine Translation for Low-Resource Languages: A Case Study on Manchu](https://mlsanity.com/paper/2502.11862/) (2025)
- [Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning](https://mlsanity.com/paper/2410.02631/) (2024)
- [SwiLTra-Bench: The Swiss Legal Translation Benchmark](https://mlsanity.com/paper/2503.01372/) (2025)
- [Conditions for Catastrophic Forgetting in Multilingual Translation](https://mlsanity.com/paper/2510.19546/) (2025)
- [Did Translation Models Get More Robust Without Anyone Even Noticing?](https://mlsanity.com/paper/2403.03923/) (2024)
- [A Preference-driven Paradigm for Enhanced Translation with Large Language Models](https://mlsanity.com/paper/2404.11288/) (2024)
- [Steering Conversational Large Language Models for Long Emotional Support Conversations](https://mlsanity.com/paper/2402.10453/) (2024)
- [Revealing the Parallel Multilingual Learning within Large Language Models](https://mlsanity.com/paper/2403.09073/) (2024)
- [Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation](https://mlsanity.com/paper/2410.07779/) (2024)
- [Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation](https://mlsanity.com/paper/2504.01919/) (2025)
- [STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models](https://mlsanity.com/paper/2505.20645/) (2025)
- [Instruction-tuned Large Language Models for Machine Translation in the Medical Domain](https://mlsanity.com/paper/2408.16440/) (2024)
- [Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis](https://mlsanity.com/paper/2409.20059/) (2024)
- [MaxSup: Overcoming Representation Collapse in Label Smoothing](https://mlsanity.com/paper/2502.15798/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are a promising avenue for machine translation (MT). However, current LLM-based MT systems are brittle: their effectiveness highly depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. Alternatives such as finetuning on translation instructions are computationally expensive and may weaken in-context learning capabilities, due to overspecialization. In this paper, we provide a closer look at this problem. We start by showing that adapter-based finetuning with LoRA matches the performance of traditional finetuning while reducing the number of training parameters by a factor of 50. This method also outperforms few-shot prompting and eliminates the need for post-processing or in-context examples. However, we show that finetuning generally degrades few-shot performance, hindering adaptation capabilities. Finally, to obtain the best of both worlds, we propose a simple approach that incorporates few-shot examples during finetuning. Experiments on 10 language pairs show that our proposed approach recovers the original few-shot capabilities while keeping the added benefits of finetuning.