Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
As agentic systems getting adopted rapidly in safety critical applications, it is vital to measure the confidence associated with the agentic actions. In comparison to the traditional machine learning systems, agentic workflows have complex failure modes with planning, tool invocation and dynamic environment interactions. In this paper, we investigate whether model's internal representations provide stronger signals of eventual task success in multi-turn agentic setups. We introduce two complementary methods: Latent Trajectory Dynamics (LTD), which summarizes changes in residual-stream representations across an interaction trajectory, and the Action Representation Probe (ARP), which predicts success from representations formed at action decisions. Across three interactive benchmarks (Bash, SQL, Python) and three model families (Qwen14B, Qwen7B, DeepSeek6.7B), our methods consistently outperform surface level generation and sequence-based calibration baselines providing a zero-overhead reliability monitor that requires neither prompt alterations nor multi-sample rollouts.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 16 of its 19 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
16 of 19 references.
Closest ancestors · 3
Same direction, newest first
- 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities4 Feb
- 2026Agentic Confidence Calibration22 Jan
- 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making20 Jun
Other references · 13
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2023The Internal State of an LLM Knows When It's Lyingcited by 659
- 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
- 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signalscited by 24
- 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
- 2026Internal Representations as Indicators of Hallucinations in Agent Tool Selectioncited by 19
- 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
- 2026Demystifying Agent Skills: Why They Work-Until They Don't
- 2026Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
- 2026From Features to Actions: Explainability in Traditional and Agentic AI Systems
- 2026The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution
- 2026From Uncertainty to Action: Learning to Steer LLM Agents
- 2026Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
- 2026Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability