# Runtime Governance for AI Agents: Policies on Paths arXiv: https://arxiv.org/abs/2603.16586 ยท HTML: https://mlsanity.com/paper/2603.16586/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 9 of its 20 references that are in the map. - First posted on arXiv 17 Mar 2026. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 9 of 9) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025) - [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024) - [Agent Drift: Quantifying Behavioral Degradation in Multi-Agent LLM Systems Over Extended Interactions](https://mlsanity.com/paper/2601.04170/) (2026) - [OpenClaw Agents on Moltbook: Risky Instruction Sharing and Norm Enforcement in an Agent-Only Social Network](https://mlsanity.com/paper/2602.02625/) (2026) - [MI9: An Integrated Runtime Governance Framework for Agentic AI](https://mlsanity.com/paper/2508.03858/) (2025) - [Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents](https://mlsanity.com/paper/2602.22302/) (2026) - [Governance-as-a-Service: A Multi-Agent Framework for AI System Compliance and Policy Enforcement](https://mlsanity.com/paper/2508.18765/) (2025) ## Built on by (most cited first, 18 of 18) - [From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AI](https://mlsanity.com/paper/2604.05229/) (2026) - [SARC: A Governance-by-Architecture Framework for Agentic AI Systems](https://mlsanity.com/paper/2605.07728/) (2026) - [PatchBoard: Schema-Grounded State Mutation for Reliable and Auditable LLM Multi-Agent Collaboration](https://mlsanity.com/paper/2605.29313/) (2026) - [Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI](https://mlsanity.com/paper/2604.19818/) (2026) - [MAGIQ: A Post-Quantum Multi-Agentic AI Governance System with Provable Security](https://mlsanity.com/paper/2605.06933/) (2026) - [Cheap Code, Costly Judgment: A Case Study on Governable Agentic Software Engineering](https://mlsanity.com/paper/2607.01087/) (2026) - [Correct Is Not Governed: Provenance Integrity in Agentic Workflows](https://mlsanity.com/paper/2608.12761/) (2026) - [The Accountability Horizon: An Impossibility Theorem for Governing Human-Agent Collectives](https://mlsanity.com/paper/2604.07778/) (2026) - [Governing What You Cannot Observe: Adaptive Runtime Governance for Autonomous AI Agents](https://mlsanity.com/paper/2604.24686/) (2026) - [Constraining Host-Level Abuse in Self-Hosted Computer-Use Agents via TEE-Backed Isolation](https://mlsanity.com/paper/2605.06393/) (2026) - [Right-to-Act: A Pre-Execution Non-Compensatory Decision Protocol for AI Systems](https://mlsanity.com/paper/2604.24153/) (2026) - [KYA: A Framework-Agnostic Trust Layer for Autonomous Systems with Verifiable Provenance and Hierarchical Policy Composition](https://mlsanity.com/paper/2605.25376/) (2026) - [The Vulnerability With No CVE: Managing Persistent Gaps Between Mandate and Authority in AI Coding Agents](https://mlsanity.com/paper/2608.05884/) (2026) - [Model-Based Agentic Software Engineering](https://mlsanity.com/paper/2608.25174/) (2026) - [Governance-as-Code: Translating EU AI Act Technical Requirements into Executable Compliance Pipelines for Generative AI Systems](https://mlsanity.com/paper/2609.20016/) (2026) - [Epi-Logic: A Conceptual Framework for Epistemic Runtime Control, Schema Validity Checking, and Controlled Accommodation in Autonomous AI Agents](https://mlsanity.com/paper/2609.24755/) (2026) - [Compliant with Local Controls, Collectively Discriminatory. A Governance Architecture for Multi-Agent AI in Regulated Finance](https://mlsanity.com/paper/2609.27994/) (2026) - [From Agent Output to Authorized Transition](https://mlsanity.com/paper/2609.28216/) (2026) ## Abstract (arXiv) AI agents -- systems that plan, reason, and act using large language models -- produce non-deterministic, path-dependent behavior that cannot be fully governed at design time, where with governed we mean striking the right balance between as high as possible successful task completion rate and the legal, data-breach, reputational and other costs associated with running agents. We argue that the execution path is the central object for effective runtime governance and formalize compliance policies as deterministic functions mapping agent identity, partial path, proposed next action, and organizational state to a policy violation probability. We show that prompt-level instructions (and "system prompts"), and static access control are special cases of this framework: the former shape the distribution over paths without actually evaluating them; the latter evaluates deterministic policies that ignore the path (i.e., these can only account for a specific subset of all possible paths). In our view, runtime evaluation is the general case, and it is necessary for any path-dependent policy. We develop the formal framework for analyzing AI agent governance, present concrete policy examples (inspired by the AI act), discuss a reference implementation, and identify open problems including risk calibration and the limits of enforced compliance.