# The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation

arXiv: https://arxiv.org/abs/2505.13090 · HTML: https://mlsanity.com/paper/2505.13090/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 26 of 26)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects](https://mlsanity.com/paper/2502.12404/) (2025)
- [Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability](https://mlsanity.com/paper/2305.10266/) (2023)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [TIM: Teaching Large Language Models to Translate with Comparison](https://mlsanity.com/paper/2307.04408/) (2023)
- [RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs](https://mlsanity.com/paper/2407.02552/) (2024)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?](https://mlsanity.com/paper/2312.12683/) (2023)
- [MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment](https://mlsanity.com/paper/2410.05873/) (2024)
- [Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention](https://mlsanity.com/paper/2410.12462/) (2024)
- [The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities](https://mlsanity.com/paper/2405.20089/) (2024)
- [Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?](https://mlsanity.com/paper/2404.14122/) (2024)
- [Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens](https://mlsanity.com/paper/2305.11550/) (2023)
- [SMOL: Professionally translated parallel data for 115 under-represented languages](https://mlsanity.com/paper/2502.12301/) (2025)
- [How Multilingual Are Large Language Models Fine-Tuned for Translation?](https://mlsanity.com/paper/2405.20512/) (2024)
- [Disentangling the Roles of Target-Side Transfer and Regularization in Multilingual Machine Translation](https://mlsanity.com/paper/2402.01772/) (2024)

## Built on by (most cited first, 1 of 1)

- [Analyzing and Improving Cross-lingual Knowledge Transfer for Machine Translation](https://mlsanity.com/paper/2601.04036/) (2026)

## Abstract (arXiv)

Prior research diverges on language diversity in LLM fine-tuning: Some studies report benefits while others find no advantages. Through controlled fine-tuning experiments across 132 translation directions, we systematically resolve these disparities. We find that expanding language diversity during fine-tuning improves translation quality for both unsupervised and -- surprisingly -- supervised pairs, despite less diverse models being fine-tuned exclusively on these supervised pairs. However, benefits plateau or decrease beyond a certain diversity threshold. We show that increased language diversity creates more language-agnostic representations. These representational adaptations help explain the improved performance in models fine-tuned with greater diversity.