Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
Chain-of-thought (CoT) prompting assumes that generated reasoning reflects a model's internal computation. We show this assumption is wrong in a specific, measurable way: models internally detect their own reasoning errors but outwardly express confidence in them. A linear probe on hidden states predicts trace correctness with 0.95 AUROC -- from the very first reasoning step (0.79) -- while verbalized confidence for wrong traces is 4.55/5, nearly identical to correct ones (4.87/5). A text-surface classifier achieves only 0.59 on the same data, confirming a 0.20-point gap invisible in the generated text. This hidden error awareness holds across three model families (Qwen, Llama, Phi), 1.5B-72B parameters, and RL-trained reasoning models (DeepSeek-R1, 0.852 AUROC). The natural question is whether this signal can fix the errors it detects. It cannot. Four interventions -- activation steering, probe-guided best-of-N, self-correction, and activation patching -- all fail; patching destroys output coherence entirely. The signal is diagnostic, not causal: a readout of computation quality, not a lever to redirect it. This delineates a boundary for mechanistic interpretability: error representations during reasoning are fundamentally different from the factual knowledge representations that prior work has successfully edited.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 16 references that are in the map. First posted on arXiv 10 May 2026.
Builds on
12 of 16 references.
Closest ancestors · 3
Same direction, newest first
- 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan
Other references · 9
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2023Let's Verify Step by Stepcited by 3,598
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2023The Internal State of an LLM Knows When It's Lyingcited by 644
- 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internalscited by 1
- 2026Why2Speak: Faithful Reasoning for Abstaining Action Policiescited by 0
- 2026Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generationcited by 0
- 2026It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectoriescited by 0
- 2026What Visual Generators Need from Teachers: Rethinking Representation Alignmentcited by 0
- 2026What Visual Generators Need from Teachers: Rethinking Representation Alignment28 Sep 2026
- 2026It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories3 Sep 2026
- 2026Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation1 Sep 2026
- 2026Why2Speak: Faithful Reasoning for Abstaining Action Policies21 Aug 2026
- 2026Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals25 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025Verifying Chain-of-Thought Reasoning via Its Computational Graph
- 2025Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning
- 2026When Chain-of-Thought Fails, the Solution Hides in the Hidden States
- 2026Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models
- 2024Understanding Hidden Computations in Chain-of-Thought Reasoning
- 2026Valid ne Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought
- 2026Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
- 2026Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning