# Security Considerations for Artificial Intelligence Agents

arXiv: https://arxiv.org/abs/2603.12230 · HTML: https://mlsanity.com/paper/2603.12230/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems](https://mlsanity.com/paper/2403.04960/) (2024)
- [Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward](https://mlsanity.com/paper/2602.12430/) (2026)
- [Attention Tracker: Detecting Prompt Injection Attacks in LLMs](https://mlsanity.com/paper/2411.00348/) (2024)
- [Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?](https://mlsanity.com/paper/2403.06833/) (2024)
- [Get my drift? Catching LLM Task Drift with Activation Deltas](https://mlsanity.com/paper/2406.00799/) (2024)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents](https://mlsanity.com/paper/2506.12104/) (2025)
- [Contextual Agent Security: A Policy for Every Purpose](https://mlsanity.com/paper/2501.17070/) (2025)
- [IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents](https://mlsanity.com/paper/2508.15310/) (2025)
- [IHEval: Evaluating Language Models on Following the Instruction Hierarchy](https://mlsanity.com/paper/2502.08745/) (2025)
- [Control Illusion: The Failure of Instruction Hierarchies in Large Language Models](https://mlsanity.com/paper/2502.15851/) (2025)
- [Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems](https://mlsanity.com/paper/2601.17548/) (2026)
- [LLM Agents Should Employ Security Principles](https://mlsanity.com/paper/2505.24019/) (2025)
- [SysBench: Can Large Language Models Follow System Messages?](https://mlsanity.com/paper/2408.10943/) (2024)
- [BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents](https://mlsanity.com/paper/2511.20597/) (2025)
- [ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking](https://mlsanity.com/paper/2512.07086/) (2025)
- [ConfusedPilot: Confused Deputy Risks in RAG-based LLMs](https://mlsanity.com/paper/2408.04870/) (2024)
- [SoK: Prompt Hacking of Large Language Models](https://mlsanity.com/paper/2410.13901/) (2024)

## Built on by (most cited first, 8 of 8)

- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agents](https://mlsanity.com/paper/2607.10487/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [Parallax: Why AI Agents That Think Must Never Act](https://mlsanity.com/paper/2604.12986/) (2026)
- [SUDP: Secret-Use Delegation Protocol for Agentic Systems](https://mlsanity.com/paper/2604.24920/) (2026)
- [Governing What the EU AI Act Excludes: Accountability for Autonomous AI Agents in Smart City Critical Infrastructure](https://mlsanity.com/paper/2605.01091/) (2026)
- [Constraining Host-Level Abuse in Self-Hosted Computer-Use Agents via TEE-Backed Isolation](https://mlsanity.com/paper/2605.06393/) (2026)
- [Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions](https://mlsanity.com/paper/2607.12406/) (2026)

## Abstract (arXiv)

This article, a lightly adapted version of Perplexity's response to NIST/CAISI Request for Information 2025-0035, details our observations and recommendations concerning the security of frontier AI agents. These insights are informed by Perplexity's experience operating general-purpose agentic systems used by millions of users and thousands of enterprises in both controlled and open-world environments. Agent architectures change core assumptions around code-data separation, authority boundaries, and execution predictability, creating new confidentiality, integrity, and availability failure modes. We map principal attack surfaces across tools, connectors, hosting boundaries, and multi-agent coordination, with particular emphasis on indirect prompt injection, confused-deputy behavior, and cascading failures in long-running workflows. We then assess current defenses as a layered stack: input-level and model-level mitigations, sandboxed execution, and deterministic policy enforcement for high-consequence actions. Finally, we identify standards and research gaps, including adaptive security benchmarks, policy models for delegation and privilege control, and guidance for secure multi-agent system design aligned with NIST risk management principles.