mlsanityData to 7 Oct 2026

Toward LLMs Beyond English-Centric Development

arXiv 2605.15613 ↗First posted 15 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

22 of 31 references.

References · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  10. 2018Learning Word Vectors for 157 Languagescited by 279
  11. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  12. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
10 more
  1. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  2. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  3. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  4. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  5. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  6. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 31
  7. 2022Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experimentscited by 23
  8. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 12
  9. 2025TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languagescited by 8
  10. 2025KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstancited by 5

Built on by

Not cited yet.