Agent Security is a Systems Problem
Not on the map. Why?
Abstract (arXiv)
We take the position that agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the system level. Through this lens, efforts to increase model robustness (the dominant viewpoint in the community) are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain. Based on our experience as cybersecurity researchers in operating systems, networks, formal methods, and adversarial machine learning, we articulate a set of core principles, grounded in decades of systems security research, that provide a foundation for designing agentic systems with predictable guarantees. As evidence, we analyze eleven representative real-world attacks on agents and discuss how systems principles, if realized, could have prevented these attacks. We also identify the research challenges that stand in the way of implementing these principles in agents.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 70 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
31 of 70 references.
References · 31
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
- 2021Causal Abstractions of Neural Networkscited by 422
- 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 151
- 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
- 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
- 2025Securing AI Agents with Information-Flow Controlcited by 121
19 more
- 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
- 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
- 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
- 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 52
- 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 49
- 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
- 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
- 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
- 2024Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization (Extended Version)cited by 34
- 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 33
- 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 21
- 2025ceLLMate: Sandboxing Browser AI Agentscited by 16
- 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
- 2025Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Treecited by 14
- 2025How Not to Detect Prompt Injections with an LLMcited by 12
- 2023Information Flow Control in Machine Learning through Modular Model Architecturecited by 11
- 2026Optimizing Agent Planning for Security and Autonomycited by 10
- 2024Validating Mechanistic Interpretations: An Axiomatic Approachcited by 4
- 2026Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuningcited by 1
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Rising From the Ashes: How Agentic AI is Unblocking Challenges in Cybersecuritycited by 0
- 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agentscited by 0
- 2026SideKernel: A Usable microVM Sandbox for AI Coding Agents on macOScited by 0
- 2026VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrailscited by 0
- 2026Kairos: Transactional Security Enforcement at the System Boundary for LLM Agentscited by 0
- 2026Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents13 Oct 2026
- 2026VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrails8 Oct 2026
- 2026SideKernel: A Usable microVM Sandbox for AI Coding Agents on macOS1 Oct 2026
- 2026Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents21 Sep 2026
- 2026Rising From the Ashes: How Agentic AI is Unblocking Challenges in Cybersecurity22 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2025Systems Security Foundations for Agentic Computing
- 2026Rethinking Agent Security as a Networking Problem
- 2025A Survey on Agentic Security: Applications, Threats and Defenses
- 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
- 2026Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
- 2025Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations
- 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack