mlsanityData to 13 Oct 2026

Changing Answer Order Can Decrease MMLU Accuracy

arXiv 2406.19470 ↗First posted 27 Jun 2024cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, we use test accuracy averaged across multiple subtasks in order to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance.

Key facts. Cited by 42 papers in the map (data through 13 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

20 of 36 references.

References · 20

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,799
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 992
  5. 2019How Can We Know What Language Models Know?cited by 958
  6. 2024Yi: Open Foundation Models by 01.AIcited by 715
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 470
  8. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 404
  9. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  10. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 266
  11. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  12. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
8 more
  1. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  2. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  3. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  4. 2019Linking artificial and human neural representations of languagecited by 37
  5. 2021Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processingcited by 30
  6. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  7. 2022Language model acceptability judgements are not always robust to contextcited by 22
  8. 2022The Curious Case of Absolute Position Embeddingscited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 16; 2025: 16; 2026: 10.

42 papers.

Sort:

By area:

  1. 2024On Memorization of Large Language Models in Logical Reasoningcited by 115
  2. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  3. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  4. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  5. 2025When2Call: When (not) to Call Toolscited by 38
  6. 2024Accuracy is Not All You Needcited by 30
  7. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  8. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 26
  9. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  10. 2026Large Language Model Reasoning Failurescited by 21
  11. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  12. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
30 more
  1. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 20
  2. 2024CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoningcited by 16
  3. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  4. 2024GenBFA: An Evolutionary Optimization Approach to Bit-Flip Attacks on LLMscited by 10
  5. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  6. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  7. 2025HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMscited by 8
  8. 2025Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests insteadcited by 7
  9. 2024Shared Imagination: LLMs Hallucinate Alikecited by 5
  10. 2025Benchmarking the Pedagogical Knowledge of Large Language Modelscited by 5
  11. 2025Memory Mosaics at scalecited by 5
  12. 2024Composable Interventions for Language Modelscited by 4
  13. 2024Questionable practices in machine learningcited by 4
  14. 2025Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacementscited by 4
  15. 2026EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languagescited by 4
  16. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  17. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
  18. 2025RoToR: Towards More Reliable Responses for Order-Invariant Inputscited by 2
  19. 2024Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Modelscited by 1
  20. 2024Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabiccited by 1
  21. 2025From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertisecited by 1
  22. 2025Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmarkcited by 1
  23. 2026Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questionscited by 1
  24. 2026From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMscited by 1
  25. 2025Metric assessment protocol in the context of answer fluctuation on MCQ taskscited by 0
  26. 2025Knowing Your Uncertainty -- On the application of LLM in social sciencescited by 0
  27. 2026Selecting Language Models for Social Science: Start Small, Start Open, and Validatecited by 0
  28. 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 0
  29. 2026Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental sciencecited by 0
  30. 2026Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Modelcited by 0