Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
LLM agents increasingly perform irreversible real-world actions, including database updates, API calls, file operations, and autonomous use of tools. However, no existing system provides formally grounded, task-level safety guarantees for the plans these agents generate. Research remains fragmented across specification, verification, and enforcement, limiting understanding of the strengths and limitations of existing approaches. To address this gap, we conducted a PRISMA 2020 systematic review of 38 studies published between 2022 and 2026 and retrieved from six academic databases. Our analysis reveals four key findings. First, the specification bottleneck remains the primary challenge: natural-language-to-formal translation achieves only 24% to 35% semantic correctness, undermining downstream verification. Second, runtime monitoring is the most mature enforcement strategy, reducing unsafe actions by 40% to 65% in controlled settings, but it does not provide complete safety guarantees. Third, the verifier tax shows that blocking 94% of unsafe actions can still result in less than 5% safe task completion because agents exploit alternative unsafe paths. Finally, no existing approach simultaneously achieves soundness, scalability, semantic correctness, and task-level safety preservation. We contribute a three-level taxonomy, a comparative analysis of existing techniques, a synthesis of evidence on the verifier tax, and a ten-problem research agenda for trustworthy agentic AI.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 62 references that are in the map. First posted on arXiv 22 Jun 2026.
Builds on
22 of 62 references.
Closest ancestors · 6
Same direction, newest first
- 2026The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents18 Mar
- 2026Towards Verifiably Safe Tool Use for LLM Agents12 Jan
- 2025Enforcing Temporal Constraints for LLM Agents25 Dec
- 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation3 Oct
- 2025AgentGuard: Runtime Verification of AI Agents28 Sep
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
Other references · 16
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
- 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
- 2024The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmapcited by 20
- 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 20
- 2025VerifyLLM: LLM-Based Pre-Execution Task Plan Verification for Robotscited by 14
- 2025Enhancing Transformation from Natural Language to Signal Temporal Logic Using LLMs with Diverse External Knowledgecited by 9
- 2025Bridging Natural Language and Formal Specification--Automated Translation of Software Requirements to LTL via Hierarchical Semantics Decomposition Using LLMscited by 9
- 2026Agentproof: Static Verification of Agent Workflow Graphscited by 9
- 2025Bridging LLM Planning Agents and Formal Methods: A Case Study in Plan Verificationcited by 7
4 more
- 2025LogicGuard: Improving Embodied LLM agents through Temporal Logic based Criticscited by 5
- 2025Plan Verification for LLM-Based Embodied Task Completion Agentscited by 4
- 2026ClarifySTL: An Interactive LLM Agent Framework for STL Transformation through Requirements Clarificationcited by 2
- 2026Logic-Based Verification of Task Allocation for LLM-Enabled Multi-Agent Manufacturing Systemscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety
- 2026AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
- 2026Securing LLM Agents Need Intent-to-Execution Integrity
- 2026ActGov: Governing LLM Agent Actions via Policy-Constrained Validation
- 2026AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systems
- 2026Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment