mlsanityData to 7 Oct 2026

Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

arXiv 2412.03304 ↗First posted 4 Dec 2024cs.CLcited by 181

Not on the map. Why?

Abstract (arXiv)

Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. These biases stem not only from differences in language but also from the cultural knowledge required to interpret questions, reducing the practical utility of translated datasets like MMLU. Furthermore, translation often introduces artefacts that can distort the meaning or clarity of questions in the target language. A common practice in multilingual evaluation is to rely on machine-translated evaluation sets, but simply translating a dataset is insufficient to address these challenges. In this work, we trace the impact of both of these issues on multilingual evaluations and ensuing model performances. Our large-scale evaluation of state-of-the-art open and proprietary models illustrates that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge. Moreover, for questions requiring geographic knowledge, an astounding 84.9% focus on either North American or European regions. Rankings of model evaluations change depending on whether they are evaluated on the full portion or the subset of questions annotated as culturally sensitive, showing the distortion to model rankings when blindly relying on translated MMLU. We release Global MMLU, an improved MMLU with evaluation coverage across 42 languages -- with improved overall quality by engaging with compensated professional and community annotators to verify translation quality while also rigorously evaluating cultural biases present in the original dataset. This comprehensive Global MMLU set also includes designated subsets labeled as culturally sensitive and culturally agnostic to allow for more holistic, complete evaluation.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 84; 2026: 96.

181 papers.

Sort:

By area:

  1. 2026MiMo-V2-Flash Technical Reportcited by 163
  2. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  3. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  4. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  5. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  6. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  7. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  8. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  9. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  10. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  11. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  12. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
38 more
  1. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  2. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  3. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  4. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  5. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 17
  6. 2025The Case for "Thick Evaluations" of Cultural Representation in AIcited by 16
  7. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  8. 2025When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracycited by 13
  9. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 13
  10. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 12
  11. 2025Evaluating Gemini in an arena for learningcited by 11
  12. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  13. 2025ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learningcited by 10
  14. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generationcited by 10
  15. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 9
  16. 2025TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languagescited by 8
  17. 2025Lugha-Llama: Adapting Large Language Models for African Languagescited by 8
  18. 2025Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMscited by 8
  19. 2026TraceRouter: Robust Safety for Large Foundation Models via Path-Level Interventioncited by 8
  20. 2025When Cultures Meet: Multicultural Text-to-Image Generationcited by 7
  21. 2025Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests insteadcited by 7
  22. 2025Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkishcited by 7
  23. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  24. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturationcited by 7
  25. 2025LLM Alignment for the Arabs: A Homogenous Culture or Diverse Ones?cited by 6
  26. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  27. 2025From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluationcited by 6
  28. 2025SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhalacited by 6
  29. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6
  30. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 6
  31. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  32. 2026Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoningcited by 6
  33. 2026GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greekcited by 6
  34. 2025KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstancited by 5
  35. 2025Pensez: Less Data, Better Reasoning -- Rethinking French LLMcited by 5
  36. 2025HyperCLOVA X THINK Technical Reportcited by 5
  37. 2025LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languagescited by 5
  38. 2025Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settingscited by 5

50 most cited of 181 All 181 (JSON)