mlsanityData to 7 Oct 2026

Can multiple-choice questions really be useful in detecting the abilities of LLMs?

arXiv 2403.17752 ↗First posted 26 Mar 2024cs.CLcited by 66

Not on the map. Why?

Abstract (arXiv)

Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) due to their simplicity and efficiency. However, there are concerns about whether MCQs can truly measure LLM's capabilities, particularly in knowledge-intensive scenarios where long-form generation (LFG) answers are required. The misalignment between the task and the evaluation method demands a thoughtful analysis of MCQ's efficacy, which we undertake in this paper by evaluating nine LLMs on four question-answering (QA) datasets in two languages: Chinese and English. We identify a significant issue: LLMs exhibit an order sensitivity in bilingual MCQs, favoring answers located at specific positions, i.e., the first position. We further quantify the gap between MCQs and long-form generation questions (LFGQs) by comparing their direct outputs, token logits, and embeddings. Our results reveal a relatively low correlation between answers from MCQs and LFGQs for identical questions. Additionally, we propose two methods to quantify the consistency and confidence of LLMs' output, which can be generalized to other QA evaluation benchmarks. Notably, our analysis challenges the idea that the higher the consistency, the greater the accuracy. We also find MCQs to be less reliable than LFGQs in terms of expected calibration error. Finally, the misalignment between MCQs and LFGQs is not only reflected in the evaluation performance but also in the embedding space. Our code and models can be accessed at https://github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

34 of 68 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
22 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  10. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  11. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  12. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  13. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  14. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  15. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  16. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  17. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 115
  18. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  19. 2023Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarkscited by 45
  20. 2022KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Modelscited by 6
  21. 2023CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Carecited by 5
  22. 2022On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifierscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 38; 2026: 14.

66 papers.

Sort:

By area:

  1. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  2. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  3. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 50
  4. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
  5. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  6. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 23
  7. 2024The Better Angels of Machine Personality: How Personality Relates to LLM Safetycited by 21
  8. 2024KGARevion: An AI Agent for Knowledge-Intensive Biomedical QAcited by 20
  9. 2025Automatic Evaluation of Healthcare LLMs Beyond Question-Answeringcited by 19
  10. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 17
  11. 2025Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluationscited by 14
  12. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
38 more
  1. 2024Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Optionscited by 11
  2. 2024Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Modelscited by 11
  3. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 9
  4. 2024In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Modelscited by 8
  5. 2025Large Language Models Often Say One Thing and Do Anothercited by 8
  6. 2024Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templatescited by 7
  7. 2025It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Educationcited by 7
  8. 2025CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensingcited by 7
  9. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
  10. 2025How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehensioncited by 6
  11. 2025Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Output Prefillingcited by 6
  12. 2024Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?cited by 5
  13. 2024Autonomous Prompt Engineering in Large Language Modelscited by 5
  14. 2025Let Androids Dream of Electric Sheep: A Human-Inspired Image Implication Understanding and Reasoning Frameworkcited by 5
  15. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Modelscited by 5
  16. 2025Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processingcited by 5
  17. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  18. 2025Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacementscited by 4
  19. 2025Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questionscited by 4
  20. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Modelscited by 4
  21. 2024Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipinocited by 3
  22. 2025On the Inevitability of Left-Leaning Political Bias in Aligned Language Modelscited by 3
  23. 2025MyCulture: Exploring Malaysia's Diverse Culture under Low-Resource Language Constraintscited by 3
  24. 2025WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thaicited by 3
  25. 2025The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarkscited by 3
  26. 2025Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variantscited by 3
  27. 2025Choices Speak Louder than Questionscited by 2
  28. 2025Patience is all you need! An agentic system for performing scientific literature reviewcited by 2
  29. 2025La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin Americacited by 2
  30. 2026RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repaircited by 2
  31. 2024Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoningcited by 1
  32. 2025OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmologycited by 1
  33. 2025What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?cited by 1
  34. 2025Impact of Comments on LLM Comprehension of Legacy Codecited by 1
  35. 2025Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?cited by 1
  36. 2025IndiCASA: A Dataset and Bias Evaluation Framework in LLMs Using Contrastive Embedding Similarity in the Indian Contextcited by 1
  37. 2025Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Modelscited by 1
  38. 2025Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choicescited by 1

50 most cited of 66 All 66 (JSON)