mlsanityData to 14 Oct 2026

MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation

arXiv 2503.10497 ↗First posted 13 Mar 2025cs.CLcited by 79

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-linguistic reasoning abilities. This dual limitation makes it challenging to comprehensively assess LLMs' performance in the multilingual setting. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons. Additionally, to meet efficient evaluation needs, we provide a lite version containing 658 questions per language. To ensure the high quality of MMLU-ProX, we employ a rigorous development process that involves multiple powerful LLMs for translation, followed by expert review to ensure accurate expression, consistent terminology, and cultural relevance. Building on this, we systematically evaluate 36 state-of-the-art LLMs, including reasoning-enhanced and multilingual-optimized LLMs. The results reveal significant disparities in the multilingual capabilities of LLMs: While they perform well in high-resource languages, their performance declines markedly in low-resource languages, with gaps of up to 24.3%. Through MMLU-ProX, we aim to advance the development of more inclusive AI systems and promote equitable access to technology across global contexts.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 44 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

30 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning17 Feb
  2. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?20 Aug
  3. 2023Do Multilingual Language Models Think Better in English?2 Aug

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
15 more
  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  2. 2024Phi-4 Technical Reportcited by 711
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  4. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  5. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  6. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  7. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  8. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  9. 2025WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialectscited by 79
  10. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  11. 2024Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performancecited by 41
  12. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  13. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkishcited by 32
  14. 2023Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Conceptscited by 11
  15. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 48.

79 papers.

Sort:

By area:

  1. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  2. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  3. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  4. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  5. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  6. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMscited by 16
  7. 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
  8. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 15
  9. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?cited by 12
  10. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Culturescited by 10
  11. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 10
  12. 2026EuroLLM-22B: Technical Reportcited by 9
38 more
  1. 2025Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMscited by 8
  2. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 7
  3. 2025Quantifying Language Disparities in Multilingual Large Language Modelscited by 7
  4. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6
  5. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  6. 2025Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerationscited by 5
  7. 2026ORCH: many analyses, one merge-a deterministic multi-agent orchestrator for discrete-choice reasoning with EMA-guided routingcited by 5
  8. 2026R3S: Refining and Recovering Reinforcement Signals for Multilingual Understanding and Reasoningcited by 5
  9. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Modelcited by 4
  10. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 4
  11. 2026The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languagescited by 4
  12. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
  13. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  14. 2026LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidancecited by 3
  15. 2025Routesplain: Towards Faithful and Intervenable Routing for Software-related Taskscited by 2
  16. 2025Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkishcited by 2
  17. 2026Med-CoReasoner: Reducing Language Disparities in Medical Reasoning via Language-Informed Co-Reasoningcited by 2
  18. 2026Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Modelscited by 2
  19. 2026Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switchcited by 2
  20. 2026Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Pluscited by 2
  21. 2026Efficient Multilingual Reasoning Transfer via Progressive Code-Switchingcited by 2
  22. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agentscited by 2
  23. 2025A Metric for MLLM Alignment in Large-scale Recommendationcited by 1
  24. 2025Do LLMs exhibit the same commonsense capabilities across languages?cited by 1
  25. 2025ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Modelscited by 1
  26. 2025Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practicescited by 1
  27. 2025The truth is no diaper: Human and AI-generated associations to emotional wordscited by 1
  28. 2026Language Models Entangle Language and Culturecited by 1
  29. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  30. 2026Less Languages, Less Tokens: An Efficient Unified Logic Cross-lingual Chain-of-Thought Reasoning Frameworkcited by 1
  31. 2026The Language of the Question Selects the Market: Query Language and Exit IP as Separable Factors in Commercial Recommendations from a Generative Search Interfacecited by 1
  32. 2025Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMscited by 0
  33. 2025Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Modelscited by 0
  34. 2025ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generationcited by 0
  35. 2025OpenJAI-v1.0: An Open Thai Large Language Modelcited by 0
  36. 2025Phishing Email Detection Using Large Language Modelscited by 0
  37. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statementscited by 0
  38. 2026BYOL: Bring Your Own Language Into LLMscited by 0

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  2. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
  3. 2024MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
  4. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
  5. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
  6. 2024Multilingual Large Language Models: A Systematic Survey
  7. 2026Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
  8. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models