mlsanityData to 7 Oct 2026

Choices Speak Louder than Questions

arXiv 2502.18798 ↗First posted 26 Feb 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent findings raise concerns about whether the evaluation of Multiple-Choice Question Answering (MCQA) accurately reflects the comprehension abilities of large language models. This paper explores the concept of choice sensitivity, which refers to the tendency for model decisions to be more influenced by the answer options than by a genuine understanding of the question. We introduce a new scoring method called Normalized Probability Shift by the Question (NPSQ), designed to isolate the impact of the question itself and provide a more reliable assessment of comprehension. Through experiments involving various input formats, including cloze, symbols, and hybrid formats, we find that traditional scoring methods - such as those based on log-likelihood or its length-normalized variant - are vulnerable to superficial characteristics of the answer choices. In contrast, NPSQ remains stable even when modifications are made to the answer options.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

38 of 43 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2024Mixtral of Expertscited by 1,785
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
26 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  6. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  7. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  9. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  10. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  11. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  12. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  13. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  14. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  15. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  16. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  17. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
  18. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  19. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  20. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 48
  21. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 43
  22. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 32
  23. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  24. 2024How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgmentscited by 14
  25. 2024Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?cited by 12
  26. 2022Partial-input baselines show that NLI models can ignore context, but they don'tcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarkscited by 1
  2. 2026Models Know Models Best: Evaluation via Model-Preferred Formatscited by 0