mlsanityData to 14 Oct 2026

Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices

arXiv 2511.21860 ↗First posted 26 Nov 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

In this work we present the Consistency-Rebalanced Accuracy (CoRA) metric, improving the reliability of Large Language Model (LLM) scores computed on multiple choice (MC) benchmarks. Our metric explores the response consistency of the LLMs, taking advantage of synthetically-generated questions with altered answer choices. With two intermediate scores, i.e. Bare-Minimum-Consistency Accuracy (BMCA) and Consistency Index (CI), CoRA is computed by adjusting the multiple-choice question answering (MCQA) scores to better reflect the level of consistency of the LLM. We present evaluations in different benchmarks using diverse LLMs, and not only demonstrate that LLMs can present low response consistency even when they present high MCQA scores, but also that CoRA can successfully scale down the scores of inconsistent models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

19 of 31 references.

References · 19

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  6. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  7. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  8. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  9. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  10. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  11. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  12. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
7 more
  1. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69
  2. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Modelscited by 34
  3. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 33
  4. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 25
  5. 2023S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Modelscited by 18
  6. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehensioncited by 13
  7. 2024A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarioscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Improving Cross-Format Robustness in Language Models with Multi-Format Trainingcited by 2