mlsanityData to 14 Oct 2026

Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review Budgets

arXiv 2609.07095 ↗First posted 7 Sep 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM assistants often produce more answers than humans can review before users see them. Most evaluations ask whether an answer is wrong, unsupported, or low-confidence. Bounded review budgets instead ask which answers should be checked first under a fixed review budget. Risk alone is not enough: a high-risk answer may be hard to repair, while a moderately risky answer may be directly correctable from available evidence. For generated-answer evaluation, we model review prioritization as exposure reduction, where review value combines estimated wrongness, intervention affordance, impact, and cost. We evaluate review queues with Wrong-Answer Exposure Ratio (WAER), the fraction of wrong answers left unreviewed, and post-repair residual exposure (PRRE), the fraction still exposed after deterministic benchmark-supported repairs. PRRE uses repairability rules that do not numerically reuse the affordance scores used for ranking. On a 720-item TAT-QA/SciFact stress benchmark, review-value ranking keeps answer-level WAER nearly unchanged at 20% budget (0.605 vs. 0.600) but lowers PRRE from 0.881 to 0.716. These results show that trustworthy LLM evaluation should measure not only error detection, but also how limited review capacity reduces exposed wrong answers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

20 of 33 references.

References · 20

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  8. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  10. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  11. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  12. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
8 more
  1. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  2. 2020Selective Question Answering under Domain Shiftcited by 238
  3. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  4. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 162
  5. 2020Learning to Complement Humanscited by 124
  6. 2021Differentiable Learning Under Triagecited by 76
  7. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
  8. 2020Classification Under Human Assistancecited by 42

Built on by

Not cited yet.