mlsanityData to 7 Oct 2026

Are We Done with MMLU?

arXiv 2406.04127 ↗First posted 6 Jun 2024cs.CL cs.AIcited by 178

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

32 of 65 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
20 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  4. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  5. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2019Human uncertainty makes classification more robustcited by 231
  8. 2024Capabilities of Gemini Models in Medicinecited by 231
  9. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  10. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  11. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  12. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
  13. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  14. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 71
  15. 2022A Survey on Programmatic Weak Supervisioncited by 66
  16. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  17. 2017The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Taskcited by 39
  18. 2021How well do you know your summarization datasets?cited by 16
  19. 2021Toward Annotator Group Bias in Crowdsourcingcited by 14
  20. 2022Weak Supervision for Affordable Modeling of Electrocardiogram Datacited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 63; 2026: 93.

178 papers.

Sort:

By area:

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  4. 2024DeepSeek-V3 Technical Reportcited by 724
  5. 2025Kimi K2: Open Agentic Intelligencecited by 523
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  7. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  8. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  9. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  10. 2026Ministral 3cited by 88
  11. 2024Non-Determinism of "Deterministic" LLM Settingscited by 84
  12. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
38 more
  1. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  2. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61
  3. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  4. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  5. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  6. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  7. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  8. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  9. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  10. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  11. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  12. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  13. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  14. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 28
  15. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  16. 2025Let LRMs Break Free from Overthinking via Self-Braking Tuningcited by 27
  17. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  18. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  19. 2026Trust Region On-Policy Distillationcited by 26
  20. 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 25
  21. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  22. 2025Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUscited by 22
  23. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 22
  24. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  25. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
  26. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 21
  27. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  28. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  29. 2024CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoningcited by 16
  30. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 16
  31. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 15
  32. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  33. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theorycited by 14
  34. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUscited by 13
  35. 2026BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarkscited by 13
  36. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  37. 2024Chasing Random: Instruction Selection Strategies Fail to Generalizecited by 12
  38. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12

50 most cited of 178 All 178 (JSON)