# Formal Policy Enforcement for Real-World Agentic Systems

arXiv: https://arxiv.org/abs/2602.16708 · HTML: https://mlsanity.com/paper/2602.16708/

## Key facts

- Cited by 31 papers in the map (data through 7 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 24 of 24)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2504.11358/) (2025)
- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [On Prompt-Driven Safeguarding for Large Language Models](https://mlsanity.com/paper/2401.18018/) (2024)
- [Securing AI Agents with Information-Flow Control](https://mlsanity.com/paper/2505.23643/) (2025)
- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization (Extended Version)](https://mlsanity.com/paper/2403.04651/) (2024)
- [A Critical Evaluation of Defenses against Prompt Injection Attacks](https://mlsanity.com/paper/2505.18333/) (2025)
- [Practical Whole-System Provenance Capture](https://mlsanity.com/paper/1711.05296/) (2017)
- [Policy-as-Prompt: Turning AI Governance Rules into Guardrails for AI Agents](https://mlsanity.com/paper/2509.23994/) (2025)
- [MALADE: Orchestration of LLM-powered Agents with Retrieval Augmented Generation for Pharmacovigilance](https://mlsanity.com/paper/2408.01869/) (2024)
- [How Not to Detect Prompt Injections with an LLM](https://mlsanity.com/paper/2507.05630/) (2025)

## Built on by (most cited first, 30 of 31)

- [A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle](https://mlsanity.com/paper/2604.16548/) (2026)
- [Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents](https://mlsanity.com/paper/2603.20953/) (2026)
- [Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents](https://mlsanity.com/paper/2607.07405/) (2026)
- [Context-to-Execution Integrity for LLM Agents](https://mlsanity.com/paper/2607.06000/) (2026)
- [Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents](https://mlsanity.com/paper/2606.22504/) (2026)
- [CONTINUITY: Security-Context Contracts for Composable LLM Agent Controls](https://mlsanity.com/paper/2609.05269/) (2026)
- [Agent Security is a Systems Problem](https://mlsanity.com/paper/2605.18991/) (2026)
- [SecureClaw: Clawing Back Control of LLM Agents](https://mlsanity.com/paper/2606.09549/) (2026)
- [Janus: a Playground for User-Involved Agentic Permission Management](https://mlsanity.com/paper/2607.01510/) (2026)
- [How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement](https://mlsanity.com/paper/2607.13718/) (2026)
- [CatchBench: When Can an Agent Failure Be Caught?](https://mlsanity.com/paper/2608.22808/) (2026)
- [AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systems](https://mlsanity.com/paper/2608.22868/) (2026)
- [Causality Laundering: Denial-Feedback Leakage in Tool-Calling LLM Agents](https://mlsanity.com/paper/2604.04035/) (2026)
- [PolicyBank: Evolving Policy Understanding for LLM Agents](https://mlsanity.com/paper/2604.15505/) (2026)
- [Owner-Harm: A Missing Threat Model for AI Agent Safety](https://mlsanity.com/paper/2604.18658/) (2026)
- [Constraining Host-Level Abuse in Self-Hosted Computer-Use Agents via TEE-Backed Isolation](https://mlsanity.com/paper/2605.06393/) (2026)
- [Prompts Don't Protect: Architectural Enforcement via MCP Proxy for LLM Tool Access Control](https://mlsanity.com/paper/2605.18414/) (2026)
- [Efficient and Sound Probabilistic Verification for AI Agents](https://mlsanity.com/paper/2606.20510/) (2026)
- [Proof-of-Execution Memory: Defending LLM Agents Against Forged-Reasoning Attacks by Verifying What Actually Happened](https://mlsanity.com/paper/2608.16032/) (2026)
- [The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities](https://mlsanity.com/paper/2607.05743/) (2026)
- [Verification of Adaptive Agentic Controllers through Finite Rule Revision](https://mlsanity.com/paper/2607.09770/) (2026)
- [Delegation Without Trust: An Empirical Gap Analysis of Identity, Authorization, and Runtime Governance in Multi-Agent LLM Systems](https://mlsanity.com/paper/2609.00267/) (2026)
- [Where Reliability Lives: Experimental Localisation of Behavioural Properties in an Agent System](https://mlsanity.com/paper/2609.03192/) (2026)
- [Versioned Transitive Dependency-Closure Binding and Operation-Time Effect Governance for Agent Skills: ClosureBound](https://mlsanity.com/paper/2609.05920/) (2026)
- [How Strongly Should Task State Influence an LLM Agent?](https://mlsanity.com/paper/2609.25686/) (2026)
- [Authorization Closure Graph: Minimal Repair for LLM Agents with Evolving User Instructions](https://mlsanity.com/paper/2609.32428/) (2026)
- [Certified Multi-Source Integrity for Structured Agent Actions](https://mlsanity.com/paper/2609.34245/) (2026)
- [Audience-Bound Persistent Memory: Authorization Across the Memory Lifecycle](https://mlsanity.com/paper/2609.36373/) (2026)
- [Sapien: A Stateful Policy Engine for Autonomous AI Agents](https://mlsanity.com/paper/2610.00797/) (2026)
- [PACE: Provenance-Aware Capability Enforcement for Tool-Using LLM Agents](https://mlsanity.com/paper/2610.01349/) (2026)

## Abstract (arXiv)

Security policy enforcement in contemporary agentic systems predominantly consists of embedding natural-language policies within an agent's system prompt and delegating compliance to the agent's reasoning. This approach admits no formal enforcement guarantee and cannot express policies whose satisfaction depends on the causal history of an execution, a gap that becomes acute in multi-agent systems, where enforcement must reason across agents. We argue that policy enforcement in agentic systems is most naturally understood as a cross-cutting concern, and propose a framework grounded in aspect-oriented programming that specifies policies independent of the agent's reasoning and enforces them at every policy-relevant decision. Policies are written in Datalog over a set of abstract predicates describing the execution context, an observability service governed by a formal assume/guarantee contract maintains these predicates, and a reference monitor consults the policy at each action to produce an enforcement decision. When the environment contract holds, enforcement decisions coincide with the policy's intended semantics. We adopt Datalog as the policy language, a natural fit because it supports declarative rule specification, admits recursion for policies over transitive relationships, and yields deterministic enforcement. Datalog further admits tractable static analyses for contradiction, redundancy, subsumption, and conditional reachability, enabling authors to verify policy intent and surface ambiguities inherent in natural-language specifications. We realize the framework in FORGE, which enforces policies over agentic deployments without modification to the underlying agents. We evaluate FORGE on three case studies: information flow policies for prompt injection defense, approval workflows in a multi-agent pharmacovigilance system, and organizational policies for customer service.