mlsanityData to 14 Oct 2026

Metric assessment protocol in the context of answer fluctuation on MCQ tasks

arXiv 2507.15581 ↗First posted 21 Jul 2025cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Using multiple-choice questions (MCQs) has become a standard for assessing LLM capabilities efficiently. A variety of metrics can be employed for this task. However, previous research has not conducted a thorough assessment of them. At the same time, MCQ evaluation suffers from answer fluctuation: models produce different results given slight changes in prompts. We suggest a metric assessment protocol in which evaluation methodologies are analyzed through their connection with fluctuation rates, as well as original performance. Our results show that there is a strong link between existing metrics and the answer changing, even when computed without any additional prompt variants. A novel metric, worst accuracy, demonstrates the highest association on the protocol.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

28 of 52 references.

References · 28

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  11. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
16 more
  1. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  2. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  3. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  4. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  5. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  6. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  7. 2018BLEU is Not Suitable for the Evaluation of Text Simplificationcited by 78
  8. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  9. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  10. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  11. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  12. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 31
  13. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 25
  14. 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
  15. 2023Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wordingcited by 13
  16. 2024Probabilistic Consensus through Ensemble Validation: A Framework for LLM Reliabilitycited by 9

Built on by

Not cited yet.