# SAGA: A Security Architecture for Governing AI Agentic Systems

arXiv: https://arxiv.org/abs/2504.21034 · HTML: https://mlsanity.com/paper/2504.21034/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 83 references that are in the map.
- First posted on arXiv 27 Apr 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 28 of 28)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast](https://mlsanity.com/paper/2402.08567/) (2024)
- [Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems](https://mlsanity.com/paper/2410.07283/) (2024)
- [Red-Teaming LLM Multi-Agent Systems via Communication Attacks](https://mlsanity.com/paper/2502.14847/) (2025)
- [AirGapAgent: Protecting Privacy-Conscious Conversational Agents](https://mlsanity.com/paper/2405.05175/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities](https://mlsanity.com/paper/2407.07791/) (2024)
- [A Survey of LLM-based Agents in Medicine: How far are we from Baymax?](https://mlsanity.com/paper/2502.11211/) (2025)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Authenticated Delegation and Authorized AI Agents](https://mlsanity.com/paper/2501.09674/) (2025)
- [Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2502.14321/) (2025)
- [Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications](https://mlsanity.com/paper/2403.02817/) (2024)
- [MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debate](https://mlsanity.com/paper/2406.14711/) (2024)
- [Multi-Agent Systems Execute Arbitrary Malicious Code](https://mlsanity.com/paper/2503.12188/) (2025)
- [Infrastructure for AI Agents](https://mlsanity.com/paper/2501.10114/) (2025)
- [textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks](https://mlsanity.com/paper/2504.00218/) (2025)
- [IDs for AI Systems](https://mlsanity.com/paper/2406.12137/) (2024)
- [Frontier AI systems have surpassed the self-replicating red line](https://mlsanity.com/paper/2412.12140/) (2024)
- [FinRobot: AI Agent for Equity Research and Valuation with Large Language Models](https://mlsanity.com/paper/2411.08804/) (2024)
- [Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environments](https://mlsanity.com/paper/2409.11276/) (2024)
- [Sharding Distributed Databases: A Critical Review](https://mlsanity.com/paper/2404.04384/) (2024)
- [Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testing](https://mlsanity.com/paper/2502.15506/) (2025)

## Built on by (most cited first, 30 of 45)

- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directions](https://mlsanity.com/paper/2510.25445/) (2025)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes](https://mlsanity.com/paper/2601.05293/) (2026)
- [Systems Security Foundations for Agentic Computing](https://mlsanity.com/paper/2512.01295/) (2025)
- [Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents](https://mlsanity.com/paper/2606.03895/) (2026)
- [SoK: Security of Autonomous LLM Agents in Agentic Commerce](https://mlsanity.com/paper/2604.15367/) (2026)
- [Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents](https://mlsanity.com/paper/2606.22504/) (2026)
- [SMCP: Secure Model Context Protocol](https://mlsanity.com/paper/2602.01129/) (2026)
- [Agent Security is a Systems Problem](https://mlsanity.com/paper/2605.18991/) (2026)
- [Overlaying Governance: A Compositional Authorization Framework for Delegation and Scope in Agentic AI](https://mlsanity.com/paper/2606.03518/) (2026)
- [Deontic Policies for Runtime Governance of Agentic AI Systems](https://mlsanity.com/paper/2606.19464/) (2026)
- [Intent-Governed Tool Authorization for AI Agents](https://mlsanity.com/paper/2606.22916/) (2026)
- [AID-Guard: Stateful Authorization for Delegated Agent Effects](https://mlsanity.com/paper/2608.21159/) (2026)
- [Faramesh: A Protocol-Agnostic Execution Control Plane for Autonomous Agent Systems](https://mlsanity.com/paper/2601.17744/) (2026)
- [Tracking Capabilities for Safer Agents](https://mlsanity.com/paper/2603.00991/) (2026)
- [An AI Agent Execution Environment to Safeguard User Data](https://mlsanity.com/paper/2604.19657/) (2026)
- [Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systems](https://mlsanity.com/paper/2605.27628/) (2026)
- [Fortifying the Agentic Web: A Unified Zero-Trust Architecture Against Logic-layer Threats](https://mlsanity.com/paper/2508.12259/) (2025)
- [Agents at Risk: How Users Unwittingly Undermine LLM Safety](https://mlsanity.com/paper/2601.10758/) (2026)
- [Agentic AI for Autonomous Defense in Software Supply Chain Security: Beyond Provenance to Vulnerability Mitigation](https://mlsanity.com/paper/2512.23480/) (2025)
- [AgenticCyOps: Securing Multi-Agentic AI Integration in Enterprise Cyber Operations](https://mlsanity.com/paper/2603.09134/) (2026)
- [SUDP: Secret-Use Delegation Protocol for Agentic Systems](https://mlsanity.com/paper/2604.24920/) (2026)
- [MAGIQ: A Post-Quantum Multi-Agentic AI Governance System with Provable Security](https://mlsanity.com/paper/2605.06933/) (2026)
- [Attacks and Mitigations for Distributed Governance of Agentic AI under Byzantine Adversaries](https://mlsanity.com/paper/2605.12364/) (2026)
- [MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents](https://mlsanity.com/paper/2512.11147/) (2025)
- [Like a Hammer, It Can Build, It Can Break: Large Language Model Uses, Perceptions, and Adoption in Cybersecurity Operations on Reddit](https://mlsanity.com/paper/2604.09998/) (2026)
- [Bounded Autonomy for Enterprise AI: Typed Action Contracts and Consumer-Side Execution](https://mlsanity.com/paper/2604.14723/) (2026)
- [Governing What the EU AI Act Excludes: Accountability for Autonomous AI Agents in Smart City Critical Infrastructure](https://mlsanity.com/paper/2605.01091/) (2026)

## Abstract (arXiv)

Large Language Model (LLM)-based agents increasingly interact, collaborate, and delegate tasks to one another autonomously with minimal human interaction. Industry guidelines for agentic system governance emphasize the need for users to maintain comprehensive control over their agents, mitigating potential damage from malicious agents. Several proposed agentic system designs address agent identity, authorization, and delegation, but remain purely theoretical, without concrete implementation and evaluation. Most importantly, they do not provide user-controlled agent management. To address this gap, we propose SAGA, a scalable Security Architecture for Governing Agentic systems, that offers user oversight over their agents' lifecycle. In our design, users register their agents with a central entity, the Provider, that maintains agent contact information, user-defined access control policies, and helps agents enforce these policies on inter-agent communication. We introduce a cryptographic mechanism for deriving access control tokens, that offers fine-grained control over an agent's interaction with other agents, providing formal security guarantees. We evaluate SAGA on several agentic tasks, using agents in different geolocations, and multiple on-device and cloud LLMs, demonstrating minimal performance overhead with no impact on underlying task utility in a wide range of conditions. Our architecture enables secure and trustworthy deployment of autonomous agents, accelerating the responsible adoption of this technology in sensitive environments.