The Knowing-Saying Gap: When Probes See Errors that Confidence Misses
Not on the map. Why?
Abstract (arXiv)
Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction. The result is a dissociation with direct implications for deployment monitoring. Across multi-hop arithmetic chains, probes that detect corruption turn out to be uninformative about final answer correctness; models forced into structured confidence formats collapse to two values with indistinguishable error rates; and probe persistence across hops fails to separate correct from incorrect outcomes, refuting our pre-registered "persistence beats peak" hypothesis. This pattern of knowing but not saying generalises across model families including reasoning models. As a real-time monitor, probe-based interventions are sharply model and error-type dependent: branch-and-pick is net-positive across models and uniquely non-breaking on Llama-3.1-8B (4 rescued, 0 broken), while reprompt and replace-prior break correct traces at roughly the rate they rescue wrong ones. Probe-based monitoring is a necessary complement to verbalised confidence, but no single intervention dominates, and the deployable answer is model-aware, error-type-aware routing.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 21 Jul 2026.
Builds on
18 of 29 references.
References · 18
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Let's Verify Step by Stepcited by 3,598
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
6 more
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
- 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Closing the Confidence-Faithfulness Gap in Large Language Models
- 2026Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)
- 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents
- 2026The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs
- 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
- 2025Linear probes rely on textual evidence: Results from leakage mitigation studies in language models
- 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages
- 2026Grounding Without Corrective Control: Truth-Tracking Profiles for Large Language Models