# Towards Verifiably Safe Tool Use for LLM Agents

arXiv: https://arxiv.org/abs/2601.08012 · HTML: https://mlsanity.com/paper/2601.08012/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 20 of 20)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems](https://mlsanity.com/paper/2403.04960/) (2024)
- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning](https://mlsanity.com/paper/2406.09187/) (2024)
- [Securing AI Agents with Information-Flow Control](https://mlsanity.com/paper/2505.23643/) (2025)
- [Attention Tracker: Detecting Prompt Injection Attacks in LLMs](https://mlsanity.com/paper/2411.00348/) (2024)
- [CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments](https://mlsanity.com/paper/2411.02305/) (2024)
- [RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage](https://mlsanity.com/paper/2502.08966/) (2025)
- [TrustAgent: Towards Safe and Trustworthy LLM-based Agents](https://mlsanity.com/paper/2402.01586/) (2024)
- [PromptShield: Deployable Detection for Prompt Injection Attacks](https://mlsanity.com/paper/2501.15145/) (2025)
- [Levels of Autonomy for AI Agents](https://mlsanity.com/paper/2506.12469/) (2025)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems](https://mlsanity.com/paper/2502.07974/) (2025)
- [Systematic Hazard Analysis for Frontier AI using STPA](https://mlsanity.com/paper/2506.01782/) (2025)
- [Procedural Adherence and Interpretability Through Neuro-Symbolic Generative Agents](https://mlsanity.com/paper/2402.16905/) (2024)
- [A comprehensive safety engineering approach for software-intensive systems based on STPA](https://mlsanity.com/paper/1612.03109/) (2016)
- [Toward user-centric feature composition for the Internet of Things](https://mlsanity.com/paper/1510.06714/) (2015)

## Built on by (most cited first, 30 of 36)

- [Formal Analysis and Supply Chain Security for Agentic AI Skills](https://mlsanity.com/paper/2603.00195/) (2026)
- [OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agents](https://mlsanity.com/paper/2603.11853/) (2026)
- [The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents](https://mlsanity.com/paper/2603.19328/) (2026)
- [Reframing LLM Agent Security as an Agent-Human Interaction Problem](https://mlsanity.com/paper/2605.24309/) (2026)
- [TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling](https://mlsanity.com/paper/2605.27690/) (2026)
- [The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies](https://mlsanity.com/paper/2602.09877/) (2026)
- [Sovereign Agentic Loops: Decoupling AI Reasoning from Execution in Real-World Systems](https://mlsanity.com/paper/2604.22136/) (2026)
- [Tracking Capabilities for Safer Agents](https://mlsanity.com/paper/2603.00991/) (2026)
- [STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systems](https://mlsanity.com/paper/2604.10286/) (2026)
- [Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes](https://mlsanity.com/paper/2605.00424/) (2026)
- [Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Libraries](https://mlsanity.com/paper/2605.10990/) (2026)
- [Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety](https://mlsanity.com/paper/2605.22643/) (2026)
- [Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving](https://mlsanity.com/paper/2604.01483/) (2026)
- [Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents](https://mlsanity.com/paper/2604.15579/) (2026)
- [Explanation-Bound Tool Execution for AI Agents: Server-Verified Action Claims Without Trusting Model Rationales](https://mlsanity.com/paper/2607.25364/) (2026)
- [Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement](https://mlsanity.com/paper/2608.14590/) (2026)
- [From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI Agents](https://mlsanity.com/paper/2603.07496/) (2026)
- [Owner-Harm: A Missing Threat Model for AI Agent Safety](https://mlsanity.com/paper/2604.18658/) (2026)
- [Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory](https://mlsanity.com/paper/2606.06523/) (2026)
- [Beyond Locality: Feature-Boundary Auditing for OS-Integrated AI](https://mlsanity.com/paper/2606.10173/) (2026)
- [Managed Autonomy at Runtime: Gear-Based Safety and Governance for Single- and Multi-Agent Cyber-Physical Systems](https://mlsanity.com/paper/2607.00334/) (2026)
- [SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing](https://mlsanity.com/paper/2607.13594/) (2026)
- [Agentao: A Policy-Governed Runtime Harness for Embeddable Tool-Using LLM Agents](https://mlsanity.com/paper/2608.13574/) (2026)
- [KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents](https://mlsanity.com/paper/2609.03588/) (2026)
- [ATLAS-RTC: Closing the Loop on LLM Agent Output with Token-Level Runtime Control](https://mlsanity.com/paper/2603.27905/) (2026)
- [Comment and Control: Hijacking Agentic Workflows via Context-Grounded Evolution](https://mlsanity.com/paper/2605.11229/) (2026)
- [It's not the Language Model, it's the Tool: Deterministic Mediation for Scientific Workflows](https://mlsanity.com/paper/2605.13245/) (2026)
- [Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools](https://mlsanity.com/paper/2606.02483/) (2026)
- [ForesightSafety-SAGE:A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents](https://mlsanity.com/paper/2606.08531/) (2026)
- [Rethinking Agent Security as a Networking Problem](https://mlsanity.com/paper/2608.12172/) (2026)

## Abstract (arXiv)

Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents to perform complex tasks, LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift LLM-based agent safety from ad hoc reliability fixes to proactive guardrails with formal guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.