# VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation

arXiv: https://arxiv.org/abs/2510.05156 · HTML: https://mlsanity.com/paper/2510.05156/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 16 of 16)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning](https://mlsanity.com/paper/2406.09187/) (2024)
- [An LLM can Fool Itself: A Prompt-Based Adversarial Attack](https://mlsanity.com/paper/2310.13345/) (2023)
- [AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection](https://mlsanity.com/paper/2502.11448/) (2025)
- [AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems](https://mlsanity.com/paper/2408.14972/) (2024)
- [ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.02330/) (2025)

## Built on by (most cited first, 30 of 49)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents](https://mlsanity.com/paper/2602.22302/) (2026)
- [SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems](https://mlsanity.com/paper/2605.13716/) (2026)
- [CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution](https://mlsanity.com/paper/2602.07918/) (2026)
- [Systems Security Foundations for Agentic Computing](https://mlsanity.com/paper/2512.01295/) (2025)
- [SafePred: A Predictive Guardrail for Computer-Using Agents via World Models](https://mlsanity.com/paper/2602.01725/) (2026)
- [Detecting Malicious Agent Skills in the Wild using Attention](https://mlsanity.com/paper/2606.23416/) (2026)
- [From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents](https://mlsanity.com/paper/2606.05805/) (2026)
- [TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems](https://mlsanity.com/paper/2603.15408/) (2026)
- [Agent Security is a Systems Problem](https://mlsanity.com/paper/2605.18991/) (2026)
- [Deontic Policies for Runtime Governance of Agentic AI Systems](https://mlsanity.com/paper/2606.19464/) (2026)
- [Evaluating Privilege Usage of Agents with Real-World Tools](https://mlsanity.com/paper/2603.28166/) (2026)
- [SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents](https://mlsanity.com/paper/2604.25562/) (2026)
- [GuardianAgentBench: Where Agents Fail and How to Guard Them](https://mlsanity.com/paper/2607.20982/) (2026)
- [What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents](https://mlsanity.com/paper/2607.22868/) (2026)
- [Cross-Layer Semantic Flow Reconstruction for Attack Detection in Agentic Systems](https://mlsanity.com/paper/2603.04469/) (2026)
- [PolicyBank: Evolving Policy Understanding for LLM Agents](https://mlsanity.com/paper/2604.15505/) (2026)
- [Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layer](https://mlsanity.com/paper/2607.28815/) (2026)
- [Resume Means Resume: A Machine-Checked Conformance Contract for Checkpoint, Interrupt, and Resume Semantics in Workflow Persistence Layers](https://mlsanity.com/paper/2608.03836/) (2026)
- [RIVA: Leveraging LLM Agents for Reliable Configuration Drift Detection](https://mlsanity.com/paper/2603.02345/) (2026)
- [CivicShield: A Cross-Domain Defense-in-Depth Framework for Securing Government-Facing AI Chatbots Against Multi-Turn Adversarial Attacks](https://mlsanity.com/paper/2603.29062/) (2026)
- [Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving](https://mlsanity.com/paper/2604.01483/) (2026)
- [Efficient and Sound Probabilistic Verification for AI Agents](https://mlsanity.com/paper/2606.20510/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement](https://mlsanity.com/paper/2608.14590/) (2026)
- [Accelerating Scientific Research with Gemini in the Real-World](https://mlsanity.com/paper/2608.26701/) (2026)
- [When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents](https://mlsanity.com/paper/2605.06731/) (2026)
- [Containment Verification: AI Safety Guarantees Independent of Alignment](https://mlsanity.com/paper/2605.09045/) (2026)
- [Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment](https://mlsanity.com/paper/2605.18672/) (2026)
- [Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory](https://mlsanity.com/paper/2606.06523/) (2026)

## Abstract (arXiv)

The deployment of autonomous AI agents in sensitive domains, such as healthcare, introduces critical risks to safety, security, and privacy. These agents may deviate from user objectives, violate data handling policies, or be compromised by adversarial attacks. Mitigating these dangers necessitates a mechanism to formally guarantee that an agent's actions adhere to predefined safety constraints, a challenge that existing systems do not fully address. We introduce VeriGuard, a novel framework that provides formal safety guarantees for LLM-based agents through a dual-stage architecture designed for robust and verifiable correctness. The initial offline stage involves a comprehensive validation process. It begins by clarifying user intent to establish precise safety specifications. VeriGuard then synthesizes a behavioral policy and subjects it to both testing and formal verification to prove its compliance with these specifications. This iterative process refines the policy until it is deemed correct. Subsequently, the second stage provides online action monitoring, where VeriGuard operates as a runtime monitor to validate each proposed agent action against the pre-verified policy before execution. This separation of the exhaustive offline validation from the lightweight online monitoring allows formal guarantees to be practically applied, providing a robust safeguard that substantially improves the trustworthiness of LLM agents.