# Scaling Laws for Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/2302.09650 · HTML: https://mlsanity.com/paper/2302.09650/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 19 Feb 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Do Current Multi-Task Optimization Methods in Deep Learning Even Help?](https://mlsanity.com/paper/2209.11379/) (2022)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection](https://mlsanity.com/paper/1809.00068/) (2018)
- [Examining Scaling and Transfer of Language Model Architectures for Machine Translation](https://mlsanity.com/paper/2202.00528/) (2022)

## Built on by (most cited first, 27 of 27)

- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Scaling Optimal LR Across Token Horizons](https://mlsanity.com/paper/2409.19913/) (2024)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Selecting Large Language Model to Fine-tune via Rectified Scaling Law](https://mlsanity.com/paper/2402.02314/) (2024)
- [MergeBench: A Benchmark for Merging Domain-Specialized LLMs](https://mlsanity.com/paper/2505.10833/) (2025)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Scaling Laws for Multilingual Language Models](https://mlsanity.com/paper/2410.12883/) (2024)
- [ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality](https://mlsanity.com/paper/2510.22037/) (2025)
- [ModernVBERT: Towards Smaller Visual Document Retrievers](https://mlsanity.com/paper/2510.01149/) (2025)
- [Towards Neural Scaling Laws on Graphs](https://mlsanity.com/paper/2402.02054/) (2024)
- [Model Merging Scaling Laws in Large Language Models](https://mlsanity.com/paper/2509.24244/) (2025)
- [The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets](https://mlsanity.com/paper/2306.14975/) (2023)
- [On the Pareto Front of Multilingual Neural Machine Translation](https://mlsanity.com/paper/2304.03216/) (2023)
- [Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?](https://mlsanity.com/paper/2408.11243/) (2024)
- [Reranking Laws for Language Generation: A Communication-Theoretic Perspective](https://mlsanity.com/paper/2409.07131/) (2024)
- [OLMoASR: Open Models and Data for Training Robust Speech Recognition Models](https://mlsanity.com/paper/2508.20869/) (2025)
- [Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model](https://mlsanity.com/paper/2510.26622/) (2025)
- ["UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset](https://mlsanity.com/paper/2602.15210/) (2026)
- [When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale](https://mlsanity.com/paper/2305.14124/) (2023)
- [The Universal Statistical Structure and Scaling Laws of Chaos and Turbulence](https://mlsanity.com/paper/2311.01358/) (2023)
- [LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation](https://mlsanity.com/paper/2406.01441/) (2024)
- [Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining](https://mlsanity.com/paper/2509.15556/) (2025)

## Abstract (arXiv)

In this work, we provide a large-scale empirical study of the scaling properties of multilingual neural machine translation models. We examine how increases in the model size affect the model performance and investigate the role of the training mixture composition on the scaling behavior. We find that changing the weightings of the individual language pairs in the training mixture only affect the multiplicative factor of the scaling law. In particular, we observe that multilingual models trained using different mixing rates all exhibit the same scaling exponent. Through a novel joint scaling law formulation, we compute the effective number of parameters allocated to each language pair and examine the role of language similarity in the scaling behavior of our models. We find little evidence that language similarity has any impact. In contrast, the direction of the multilinguality plays a significant role, with models translating from multiple languages into English having a larger number of effective parameters per task than their reversed counterparts. Finally, we leverage our observations to predict the performance of multilingual models trained with any language weighting at any scale, significantly reducing efforts required for language balancing in large multilingual models. Our findings apply to both in-domain and out-of-domain test sets and to multiple evaluation metrics, such as ChrF and BLEURT.