# Toward LLMs Beyond English-Centric Development

arXiv: https://arxiv.org/abs/2605.15613 · HTML: https://mlsanity.com/paper/2605.15613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 15 May 2026.

## Builds on (most cited first, 22 of 22)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages](https://mlsanity.com/paper/2305.18098/) (2023)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)
- [LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs](https://mlsanity.com/paper/2407.03963/) (2024)
- [Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments](https://mlsanity.com/paper/2202.06387/) (2022)
- [Scaling Laws for Upcycling Mixture-of-Experts Language Models](https://mlsanity.com/paper/2502.03009/) (2025)
- [TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages](https://mlsanity.com/paper/2502.11020/) (2025)
- [KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan](https://mlsanity.com/paper/2502.12829/) (2025)

## Abstract (arXiv)

Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.