# Agent Security is a Systems Problem

arXiv: https://arxiv.org/abs/2605.18991 · HTML: https://mlsanity.com/paper/2605.18991/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 70 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 30 of 31)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2504.11358/) (2025)
- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [Securing AI Agents with Information-Flow Control](https://mlsanity.com/paper/2505.23643/) (2025)
- [Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?](https://mlsanity.com/paper/2403.06833/) (2024)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [ACE: A Security Architecture for LLM-Integrated App Systems](https://mlsanity.com/paper/2504.20984/) (2025)
- [PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails](https://mlsanity.com/paper/2402.15911/) (2024)
- [VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation](https://mlsanity.com/paper/2510.05156/) (2025)
- [SAGA: A Security Architecture for Governing AI Agentic Systems](https://mlsanity.com/paper/2504.21034/) (2025)
- [AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents](https://mlsanity.com/paper/2505.05849/) (2025)
- [A Critical Evaluation of Defenses against Prompt Injection Attacks](https://mlsanity.com/paper/2505.18333/) (2025)
- [Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization (Extended Version)](https://mlsanity.com/paper/2403.04651/) (2024)
- [Formal Policy Enforcement for Real-World Agentic Systems](https://mlsanity.com/paper/2602.16708/) (2026)
- [MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks](https://mlsanity.com/paper/2602.09222/) (2026)
- [ceLLMate: Sandboxing Browser AI Agents](https://mlsanity.com/paper/2512.12594/) (2025)
- [May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks](https://mlsanity.com/paper/2507.07417/) (2025)
- [Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree](https://mlsanity.com/paper/2507.14799/) (2025)
- [How Not to Detect Prompt Injections with an LLM](https://mlsanity.com/paper/2507.05630/) (2025)
- [Information Flow Control in Machine Learning through Modular Model Architecture](https://mlsanity.com/paper/2306.03235/) (2023)
- [Optimizing Agent Planning for Security and Autonomy](https://mlsanity.com/paper/2602.11416/) (2026)
- [Validating Mechanistic Interpretations: An Axiomatic Approach](https://mlsanity.com/paper/2407.13594/) (2024)

## Built on by (most cited first, 5 of 5)

- [Rising From the Ashes: How Agentic AI is Unblocking Challenges in Cybersecurity](https://mlsanity.com/paper/2606.23138/) (2026)
- [Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents](https://mlsanity.com/paper/2609.24515/) (2026)
- [SideKernel: A Usable microVM Sandbox for AI Coding Agents on macOS](https://mlsanity.com/paper/2610.02456/) (2026)
- [VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrails](https://mlsanity.com/paper/2610.12813/) (2026)
- [Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents](https://mlsanity.com/paper/2610.18457/) (2026)

## Abstract (arXiv)

We take the position that agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the system level. Through this lens, efforts to increase model robustness (the dominant viewpoint in the community) are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain. Based on our experience as cybersecurity researchers in operating systems, networks, formal methods, and adversarial machine learning, we articulate a set of core principles, grounded in decades of systems security research, that provide a foundation for designing agentic systems with predictable guarantees. As evidence, we analyze eleven representative real-world attacks on agents and discuss how systems principles, if realized, could have prevented these attacks. We also identify the research challenges that stand in the way of implementing these principles in agents.