Confidence is Not Competence
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) often exhibit a puzzling disconnect between their asserted confidence and actual problem-solving competence. We offer a mechanistic account of this decoupling by analyzing the geometry of internal states across two phases - pre-generative assessment and solution execution. A simple linear probe decodes the internal "solvability belief" of a model, revealing a well-ordered belief axis that generalizes across model families and across math, code, planning, and logic tasks. Yet, the geometries diverge - although belief is linearly decodable, the assessment manifold has high linear effective dimensionality as measured from the principal components, while the subsequent reasoning trace evolves on a much lower-dimensional manifold. This sharp reduction in geometric complexity from thought to action mechanistically explains the confidence-competence gap. Causal interventions that steer representations along the belief axis leave final solutions unchanged, indicating that linear nudges in the complex assessment space do not control the constrained dynamics of execution. We thus uncover a two-system architecture - a geometrically complex assessor feeding a geometrically simple executor. These results challenge the assumption that decodable beliefs are actionable levers, instead arguing for interventions that target the procedural dynamics of execution rather than the high-level geometry of assessment.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 23 references that are in the map. First posted on arXiv 24 Oct 2025.
Builds on
21 of 23 references.
References · 21
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023Steering Language Models With Activation Engineeringcited by 922
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2024Programming Refusal with Conditional Activation Steeringcited by 206
- 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
- 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
9 more
- 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
- 2024Style Vectors for Steering Generative Large Language Modelcited by 56
- 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 47
- 2025One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMscited by 29
- 2024Correcting Biased Centered Kernel Alignment Measures in Biological and Artificial Neural Networkscited by 18
- 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
- 2023Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generationcited by 12
- 2023The Confidence-Competence Gap in Large Language Models: A Cognitive Studycited by 9
- 2025Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosiscited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
- 2026Reasoning emerges from constrained inference manifolds in large language models
- 2026Closing the Confidence-Faithfulness Gap in Large Language Models
- 2026Reasoning Models Don't Just Think Longer, They Move Differently
- 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- 2026When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models
- 2025Do Large Language Models Know What They Are Capable Of?
- 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning