mlsanityData to 14 Oct 2026

Multiple-Choice Questions are Efficient and Robust LLM Evaluators

arXiv 2405.11966 ↗First posted 20 May 2024cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

We present GSM-MC, a multiple-choice (MC) dataset constructed by collecting answers and incorrect predictions on GSM8K from 60 open-source models. Through extensive experiments, we show that LLMs' performance on the MC version of this popular benchmark is strongly correlated with their performance on the original version and is quite robust to distractor choices and option orders, while the evaluation time is reduced by a factor of up to 30. Following similar procedures, we introduce MATH-MC, constructed from MATH, and PythonIO, a new program reasoning MC dataset constructed from HumanEval and MBPP. Experimental results indicate that LLMs' performance on these MC benchmarks leaves much room for improvement. Our data and code are available at https://github.com/Geralt-Targaryen/MC-Evaluation.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

25 of 34 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
13 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  6. 2023StarCoder: may the source be with you!cited by 799
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  10. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  11. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  12. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  13. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Unitscited by 34
  2. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  3. 2025TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understandingcited by 24
  4. 2024ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understandingcited by 15
  5. 2024Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Optionscited by 11
  6. 2025Reasoning Models are Test Exploiters: Rethinking Multiple-Choicecited by 11
  7. 2025Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelinescited by 10
  8. 2024In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Modelscited by 8
  9. 2025Memorization or Reasoning? Exploring the Idiom Understanding of LLMscited by 7
  10. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  11. 2025FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMscited by 5
  12. 2025PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguesecited by 5
16 more
  1. 2026RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Modelscited by 5
  2. 2024EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-betacited by 3
  3. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 3
  4. 2026Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QAcited by 3
  5. 2026Auditing LLM Benchmarks with Item Response Theorycited by 3
  6. 2025DIF: A Framework for Benchmarking and Verifying Implicit Bias in LLMscited by 2
  7. 2025D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Modelcited by 1
  8. 2025Impact of Comments on LLM Comprehension of Legacy Codecited by 1
  9. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  10. 2026Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectoriescited by 1
  11. 2026Overwhelmed by Choice: Studying LLM Decision Making at Scalecited by 1
  12. 2025From Understanding to Generation: An Efficient Shortcut for Evaluating Language Modelscited by 0
  13. 2025Towards Scalable Oversight via Partitioned Human Supervisioncited by 0
  14. 2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMscited by 0
  15. 2026SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptationcited by 0
  16. 2026Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Modelscited by 0