mlsanityData to 7 Oct 2026

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

arXiv 2406.01574 ↗First posted 3 Jun 2024cs.CLcited by 1,758

Not on the map. Why?

Cited by 452 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.

Key facts. Cited by 1,758 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

36 of 56 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
24 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024Mixtral of Expertscited by 1,785
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
  13. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  15. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  16. 2023Llemma: An Open Language Model For Mathematicscited by 385
  17. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  18. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  19. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  20. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  21. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  22. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  23. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  24. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 106; 2025: 775; 2026: 877.

1,758 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  10. 2024DeepSeek-V3 Technical Reportcited by 724
  11. 2025Qwen2.5-Omni Technical Reportcited by 704
  12. 2025Kimi K2: Open Agentic Intelligencecited by 523
38 more
  1. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  2. 2025Humanity's Last Examcited by 467
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  5. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  6. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  7. 2025Muon is Scalable for LLM Trainingcited by 381
  8. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  10. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  11. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  12. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  13. 2026Reinforcement Learning via Self-Distillationcited by 285
  14. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  15. 2025Learning to Reason under Off-Policy Guidancecited by 266
  16. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  17. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  18. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  19. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  20. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  21. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  22. 2025Why Language Models Hallucinatecited by 205
  23. 2025Learning to Reason without External Rewardscited by 199
  24. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  25. 2026Gemma 4 Technical Reportcited by 187
  26. 2024Are We Done with MMLU?cited by 178
  27. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  28. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  29. 2026MiMo-V2-Flash Technical Reportcited by 163
  30. 2025Qwen2.5-1M Technical Reportcited by 158
  31. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  32. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  33. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 146
  34. 2026Qwen3.5-Omni Technical Reportcited by 146
  35. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  36. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  37. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  38. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 129

50 most cited of 1,758 All 1,758 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning
  2. 2024MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models
  3. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  4. 2025BnMMLU: Measuring Massive Multitask Language Understanding in Bengali
  5. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  6. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
  7. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
  8. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches