mlsanityData to 14 Oct 2026

Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution

arXiv 2605.19228 ↗First posted 19 May 2026cs.CL cs.AI cs.ITcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. In this paper, we introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5% over answer-level feedback.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 47 of its 55 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

47 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processes25 Jul
  2. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey20 Mar
  3. 2024Learning From Correctness Without Prompting Makes LLM Efficient Reasoner28 Mar
  4. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks12 Feb
  5. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2023Teaching Large Language Models to Self-Debugcited by 988
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
30 more
  1. 2017Selective Classification for Deep Neural Networkscited by 852
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  6. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  7. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  8. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  9. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  10. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 177
  11. 2025Phi-4-reasoning Technical Reportcited by 151
  12. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  13. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
  14. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  15. 2023Selectively Answering Ambiguous Questionscited by 66
  16. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 65
  17. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 42
  18. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  19. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 25
  20. 2024LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentationcited by 23
  21. 2025Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMscited by 22
  22. 2025Uncertainty-Aware Step-wise Verification with Generative Reward Modelscited by 22
  23. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approachcited by 21
  24. 2025Uncertainty Quantification of Large Language Models through Multi-Dimensional Responsescited by 18
  25. 2024Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generationcited by 16
  26. 2025Adaptive Graph of Thoughts: Test-Time Adaptive Reasoning Unifying Chain, Tree, and Graph Structurescited by 16
  27. 2025Graph of Verification: Structured Verification of LLM Reasoning with Directed Acyclic Graphscited by 8
  28. 2026Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decompositioncited by 7
  29. 2025Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generationcited by 6
  30. 2026Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoderscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervisioncited by 1
  2. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
  2. 2025All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning
  3. 2025CER: Confidence Enhanced Reasoning in LLMs
  4. 2026EvalQReason: A Framework for Step-Level Reasoning Evaluation in Large Language Models
  5. 2026Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning
  6. 2025Confidence Improves Self-Consistency in LLMs
  7. 2026Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
  8. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs