mlsanityData to 7 Oct 2026

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards

arXiv 2402.01781 ↗First posted 1 Feb 2024cs.CL cs.AI cs.LGcited by 134

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

27 of 43 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
15 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  3. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  4. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  6. 2023Larger language models do in-context learning differentlycited by 386
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  8. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  9. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  10. 2021The Benchmark Lotterycited by 133
  11. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  12. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  13. 2018Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graphcited by 67
  14. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  15. 2023Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learningcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 49; 2026: 48.

134 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  3. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  5. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  6. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  7. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  8. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  9. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  10. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  11. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  12. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
38 more
  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  2. 2024Compression Represents Intelligence Linearlycited by 56
  3. 2024Quantifying Variance in Evaluation Benchmarkscited by 52
  4. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  5. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  6. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  7. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  8. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  9. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 33
  10. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  11. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  12. 2024ECBD: Evidence-Centered Benchmark Design for NLPcited by 27
  13. 2024Accuracy is Not All You Needcited by 27
  14. 2025Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answeringcited by 27
  15. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  16. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 24
  17. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  18. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  19. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  20. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  21. 2026Large Language Model Reasoning Failurescited by 20
  22. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  23. 2025Automatic Evaluation of Healthcare LLMs Beyond Question-Answeringcited by 19
  24. 2025SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Modelscited by 19
  25. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  26. 2024Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdoutscited by 13
  27. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 13
  28. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  29. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  30. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  31. 2024Evaluating LLM Reasoning in the Operations Research Domain with ORQAcited by 12
  32. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  33. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  34. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  35. 2024The Impact of Inference Acceleration on Bias of LLMscited by 9
  36. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  37. 2025Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?cited by 8
  38. 2025The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behaviorcited by 8

50 most cited of 134 All 134 (JSON)