mlsanityData to 14 Oct 2026

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

arXiv 2607.24268 ↗First posted 27 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 27 Jul 2026.

Builds on

24 of 28 references.

References · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
12 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  6. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  7. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  8. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 55
  9. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  10. 2024Large Language Models Can Self-Correct with Key Condition Verificationcited by 39
  11. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  12. 2026When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliabilitycited by 8

Built on by

Not cited yet.