mlsanityData to 7 Oct 2026

LLMs May Perform MCQA by Selecting the Least Incorrect Option

arXiv 2402.01349 ↗First posted 2 Feb 2024cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

In the field of NLP, Large Language Models (LLMs) have markedly enhanced performance across a variety of tasks. However, the comprehensive evaluation of LLMs remains an inevitable challenge for the community. Recently, the adoption of Multiple Choice Question Answering (MCQA) as a benchmark for assessing LLMs has gained considerable traction. However, concerns regarding the robustness of this evaluative method persist. Building upon previous discussions on the issue of textitvariability, we reveal an additional dimension of concern: LLMs may perform MCQA by selecting the least incorrect option rather than distinctly correct. This observation suggests that LLMs might regard multiple options as correct, which could undermine the reliability of MCQA as a metric for evaluating LLMs. To address this challenge, we introduce an enhanced dataset augmentation method for MCQA, termed MCQA+, to provide a more accurate reflection of the model performance, thereby highlighting the necessity for more sophisticated evaluation mechanisms in the assessment of LLM capabilities.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

18 of 26 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
6 more
  1. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  2. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  3. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  4. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  5. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  6. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  2. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  3. 2025Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answeringcited by 27
  4. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  5. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  6. 2024Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questionscited by 15
  7. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  8. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 12
  9. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  10. 2024Self-Training Meets Consistency: Improving LLMs' Reasoning with Consistency-Driven Rationale Evaluationcited by 6
  11. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Modelscited by 5
  12. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Modelscited by 4
20 more
  1. 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Modelscited by 3
  2. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  3. 2025Choices Speak Louder than Questionscited by 2
  4. 2024Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabiccited by 1
  5. 2025Impact of Comments on LLM Comprehension of Legacy Codecited by 1
  6. 2025Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choicescited by 1
  7. 2025CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variationscited by 1
  8. 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebookcited by 1
  9. 2026AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QAcited by 1
  10. 2026SCRuB: Social Concept Reasoning under Rubric-Based Evaluationcited by 1
  11. 2026Improving Cross-Format Robustness in Language Models with Multi-Format Trainingcited by 1
  12. 2026YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanesecited by 1
  13. 2025Rethinking Data: Towards Better Performing Domain-Specific Small Language Modelscited by 0
  14. 2026OI-Bench: An Option Injection Benchmark for Evaluating LLM Susceptibility to Directive Interferencecited by 0
  15. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 0
  16. 2026Rethinking Vacuity for OOD Detection in Evidential Deep Learningcited by 0
  17. 2026When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understandingcited by 0
  18. 2026Accuracy and Order Sensitivity Diverge Under Label-Free Strategiescited by 0
  19. 2026LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuancescited by 0
  20. 2026When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jevcited by 0