mlsanityData to 7 Oct 2026

TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning

arXiv 2602.11409 ↗First posted 11 Feb 2026cs.AIcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Estimating uncertainty for AI agents in real-world multi-turn tool-using interaction with humans is difficult because failures are often triggered by sparse critical episodes (e.g., looping, incoherent tool use, or user-agent miscoordination) even when local generation appears confident. Existing uncertainty proxies focus on single-shot text generation and therefore miss these trajectory-level breakdown signals. We introduce TRACER, a trajectory-level uncertainty metric for dual-control Tool-Agent-User interaction. TRACER combines content-aware surprisal with situational-awareness signals, semantic and lexical repetition, and tool-grounded coherence gaps, and aggregates them using a tail-focused risk functional with a MAX-composite step risk to surface decisive anomalies. We evaluate TRACER on τ2-bench by predicting task failure and selective task execution. To this end, TRACER improves AUROC by up to 37.1% and AUARC by up to 55% over baselines, enabling earlier and more accurate detection of uncertainty in complex conversational tool-use settings. Our code and benchmark are available at https://github.com/sinatayebati/agent-tracer.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 26 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

21 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making20 Jun
  2. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2019How Can We Know What Language Models Know?cited by 953
7 more
  1. 2017Selective Classification for Deep Neural Networkscited by 799
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  3. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  4. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  5. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  6. 2025Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Modelscited by 20
  7. 2025Calibrated Decomposition of Aleatoric and Epistemic Uncertainty in Deep Features for Inference-Time Adaptationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluationcited by 7
  2. 2026TRIAGE: Type-Routed Interventions via Aleatoric-Epistemic Gated Estimation in Robotic Manipulation and Adaptive Perception -- Don't Treat All Uncertainty the Samecited by 3
  3. 2026Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasetscited by 2
  4. 2026OTAP: Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectoriescited by 2
  5. 2026Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carriercited by 1
  6. 2026CURA: Certified Runtime Alarms for Computer-Use Agentscited by 1
  7. 2026From Sequence to Structure: Relational Uncertainty Propagation for LLM Agentscited by 0
  8. 2026FLARE: A Full-Lifecycle Dense Supervision Paradigm for Long-Horizon Coding Agents via Generative Reward Modelcited by 0
  9. 2026Self-Healing Harness for Runtime Oversight of Agent Self-Modificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GRUET: Quantifying Uncertainty of Agentic Reasoning-and-Acting Processes
  2. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents
  3. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
  4. 2026Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments
  5. 2026Proper Scoring Rules for Agentic Uncertainty Quantification
  6. 2026ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift
  7. 2026Clarify the User or Verify the World? Uncertainty Routing for Proactive Agents
  8. 2026CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty