mlsanityData to 14 Oct 2026

Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above

arXiv 2502.14127 ↗First posted 19 Feb 2025cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform. We first reveal flaws in MCQA's format, as it struggles to: 1) test generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge. We instead advocate for generative formats based on human testing, where LLMs construct and explain answers, better capturing user needs and knowledge while remaining easy to score. We then show even when MCQA is a useful format, its datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In each issue, we give fixes from education, like rubrics to guide MCQ writing; scoring methods to bridle guessing; and Item Response Theory to build harder MCQs. Lastly, we discuss LLM errors in MCQA, robustness, biases, and unfaithful explanations, showing how our prior solutions better measure or address these issues. While we do not need to desert MCQA, we encourage more efforts in refining the task based on educational testing, advancing evaluations.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 123 of its 223 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

123 of 223 references.

References · 123

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  10. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  13. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  14. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  15. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  16. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  17. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  18. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  19. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  20. 2024Alignment faking in large language modelscited by 360
  21. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  22. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  23. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  24. 2022Data Contamination: From Memorization to Exploitationcited by 236
  25. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  26. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  27. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  28. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  29. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  30. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  31. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  32. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  33. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  34. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  35. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  36. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  37. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  38. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126

50 most cited of 123 All 123 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 27.

52 papers.

Sort:

By area:

  1. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  2. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 30
  3. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  4. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  5. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  6. 2025BLAB: Brutally Long Audio Benchcited by 11
  7. 2025Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Modelscited by 8
  8. 2026Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activationscited by 8
  9. 2025Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Output Prefillingcited by 6
  10. 2025Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questionscited by 6
  11. 2025WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Modelscited by 5
  12. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Modelscited by 5
38 more
  1. 2025Robustness assessment of large audio language models in multiple-choice evaluationcited by 5
  2. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  3. 2026BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarkscited by 5
  4. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  5. 2025Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensionalcited by 3
  6. 2026Common to Whom? Regional Cultural Commonsense and LLM Bias in Indiacited by 3
  7. 2026ABCD: All Biases Come Disguisedcited by 3
  8. 2026Auditing LLM Benchmarks with Item Response Theorycited by 3
  9. 2025Choices Speak Louder than Questionscited by 2
  10. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corporacited by 2
  11. 2025Countermind: A Multi-Layered Security Architecture for Large Language Modelscited by 2
  12. 2025Solving Spatial Supersensing Without Spatial Supersensingcited by 2
  13. 2025ORCA: Open-ended Response Correctness Assessment for Audio Question Answeringcited by 2
  14. 2026Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Userscited by 2
  15. 2026Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Misscited by 2
  16. 2026Improving Cross-Format Robustness in Language Models with Multi-Format Trainingcited by 2
  17. 2025Evaluating Large Language Models for Fair and Reliable Organ Allocationcited by 1
  18. 2025Visual Compositional Tuningcited by 1
  19. 2025Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulationcited by 1
  20. 2025Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFTcited by 1
  21. 2026TS-Skill: A Benchmark for Evaluating Analytical Skills in Time-Series Question Answeringcited by 1
  22. 2026Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEvalcited by 1
  23. 2026Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?cited by 1
  24. 2026Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysiscited by 1
  25. 2026Cascaded Batch Promptingcited by 1
  26. 2025Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilitiescited by 0
  27. 2025The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcutscited by 0
  28. 2026Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluationcited by 0
  29. 2026BAGEL: Benchmarking Animal Knowledge Expertise in Language Modelscited by 0
  30. 2026Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankingscited by 0
  31. 2026At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalizationcited by 0
  32. 2026Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarkscited by 0
  33. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidencecited by 0
  34. 2026Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responsescited by 0
  35. 2026Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluationcited by 0
  36. 2026VibeJam: A User Study Platform for Web Development with Agentscited by 0
  37. 2026Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakagecited by 0
  38. 2026WordPolo: Evaluating Language Models Through Iterative Semantic Feedbackcited by 0

50 most cited of 52 All 52 (JSON)