Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
As LLMs are deployed in high-stakes settings, users must judge the correctness of individual responses, often relying on model-generated justifications such as reasoning chains or explanations. Yet, no standard measure exists for whether these justifications help users distinguish correct answers from incorrect ones. We formalize this idea as error verifiability and propose vbal, a balanced metric that measures whether justifications enable raters to accurately assess answer correctness, validated against human raters who show high agreement. We find that neither common approaches, such as post-training and model scaling, nor more targeted interventions recommended improve verifiability. We introduce two methods that succeed at improving verifiability: reflect-and-rephrase (RR) for mathematical reasoning and oracle-rephrase (OR) for factual QA, both of which improve verifiability by incorporating domain-appropriate external information. Together, our results establish error verifiability as a distinct dimension of response quality that does not emerge from accuracy improvements alone and requires dedicated, domain-aware methods to address.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 40 of its 45 references that are in the map. First posted on arXiv 6 Apr 2026.
Builds on
40 of 45 references.
Closest ancestors · 4
Same direction, newest first
- 2025Scalable Best-of-N Selection for Large Language Models via Self-Certainty25 Feb
- 2025Confidence Improves Self-Consistency in LLMs10 Feb
- 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation28 Jul
- 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
Other references · 36
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023Let's Verify Step by Stepcited by 3,598
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2020Learning to summarize from human feedbackcited by 2,562
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
- 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
- 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
24 more
- 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
- 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
- 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
- 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
- 2024What Large Language Models Know and What People Think They Knowcited by 109
- 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
- 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 80
- 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
- 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
- 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
- 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
- 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
- 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
- 2025Training Language Models to Explain Their Own Computationscited by 27
- 2024To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Modelscited by 17
- 2026Counterfactual Simulation Training for Chain-of-Thought Faithfulnesscited by 15
- 2024LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Modelscited by 14
- 2026A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behaviorcited by 12
- 2025Measuring and mitigating overreliance to build human-compatible AIcited by 9
- 2025Counterfactual Simulatability of LLM Explanations for Generation Taskscited by 7
- 2025Improving Human Verification of LLM Reasoning through Interactive Explanation Interfacescited by 7
- 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbationscited by 7
- 2026Evaluating Chain-of-Thought Reasoning through Reusability and Verifiabilitycited by 4
- 2026Mitigating Legibility Tax with Decoupled Prover-Verifier Gamescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models
- 2026Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
- 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
- 2026CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency
- 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only
- 2026Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
- 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
- 2025When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers