Progressive Autonomy as Preference Learning: A Formalization of Trust Calibration for Agentic Tool Use
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
We formalize trust calibration for agentic tool use (deciding when an automated agent's proposed action may execute autonomously versus require human approval) as a preference-learning problem. A policy gateway maintains a Gaussian-process posterior over a latent human risk-tolerance function, observed through a probit likelihood on binary approve/deny feedback, and escalates to the human exactly where the approval outcome is most uncertain. We show this is structurally an instance of Preferential Bayesian Optimization, inheriting its inference machinery (approximate Gaussian-process classification) and its sample-efficiency argument (uncertainty-targeted querying), while differing in objective: classifying an action space into allow/block/ask regions rather than optimizing a design.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 8 of its 18 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
8 of 18 references.
References · 8
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
- 2013Expectation Propagation for approximate Bayesian inferencecited by 172
- 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
- 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
- 2024Visibility into AI Agentscited by 77
- 2017Preferential Bayesian Optimizationcited by 72
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI
- 2026Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
- 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
- 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
- 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback
- 2025Predictive Preference Learning from Human Interventions
- 2026The Confidence Game: Strategic Miscalibration in Human-AI Delegation
- 2026Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL