mlsanityData to 14 Oct 2026

Design Patterns for Securing LLM Agents against Prompt Injections

arXiv 2506.08837 ↗First posted 10 Jun 2025cs.LG cs.CRcited by 73

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge. Among the most pressing threats are prompt injection attacks, which exploit the agent's resilience on natural language inputs -- an especially dangerous threat when agents are granted tool access or handle sensitive information. In this work, we propose a set of principled design patterns for building AI agents with provable resistance to prompt injection. We systematically analyze these patterns, discuss their trade-offs in terms of utility and security, and illustrate their real-world applicability through a series of case studies.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 24 of its 37 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

24 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Defeating Prompt Injections by Design24 Mar
  2. 2025Firewalls to Secure Dynamic LLM Agentic Networks3 Feb
  3. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  4. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agents8 May
  5. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems8 Mar

Other references · 19

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  7. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  8. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  9. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  11. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  12. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
7 more
  1. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 101
  2. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  3. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  4. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 72
  5. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
  6. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 42
  7. 2024Exfiltration of personal information from ChatGPT via prompt injectioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 56.

73 papers.

Sort:

By area:

  1. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 90
  2. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  3. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  4. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 18
  5. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 17
  6. 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deploymentcited by 15
  7. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  8. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 13
  9. 2025Systems Security Foundations for Agentic Computingcited by 13
  10. 2025Semantic Attacks on Tool-Augmented LLMs: Securing the Model Context Protocol Against Descriptor-Level Manipulationcited by 13
  11. 2025CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitizationcited by 12
  12. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
38 more
  1. 2025Preventing Prompt Injection with Type-Directed Privilege Separationcited by 9
  2. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 9
  3. 2026Agentic Business Process Management: A Research Manifestocited by 8
  4. 2026Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agentscited by 8
  5. 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigmscited by 7
  6. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  7. 2025A2AS: Agentic AI Runtime Security and Self-Defensecited by 5
  8. 2026Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agentscited by 5
  9. 2025Please Don't Kill My Vibe: Empowering Agents with Data Flow Controlcited by 4
  10. 2025With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systemscited by 4
  11. 2026Agentic Design Patterns: A System-Theoretic Frameworkcited by 4
  12. 2026Web Agents Should Adopt the Plan-Then-Execute Paradigmcited by 4
  13. 2025AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applicationscited by 3
  14. 2026STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systemscited by 3
  15. 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agentscited by 3
  16. 2026Janus: a Playground for User-Involved Agentic Permission Managementcited by 3
  17. 2026Untrusted Content Masking for Web Agents with Security Guaranteescited by 3
  18. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
  19. 2026Herding CATs: ALARA for Agent Harness Engineering in Portable Composable Multi-Agent Teamscited by 2
  20. 2026The Insurability Frontier of AI Risk: Mapping Threats to Affirmative Coverage, Silent Exposures, and Exclusionscited by 2
  21. 2026Hallucination as Exploit: Evidence-Carrying Multimodal Agentscited by 2
  22. 2026Agentic-J: An AI Agent for Biological Microscopy Image Analysiscited by 2
  23. 2026Agent Security Needs Redefinition through a Holistic Frameworkcited by 2
  24. 2026ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injectioncited by 2
  25. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  26. 2025BinaryShield: Cross-Service Threat Intelligence in LLM Services using Privacy-Preserving Fingerprintscited by 1
  27. 2025Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeedcited by 1
  28. 2025MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agentscited by 1
  29. 2026IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injectioncited by 1
  30. 2026LACUNA: Safe Agents as Recursive Program Holescited by 1
  31. 2026Game-Theoretic Multi-Agent Control for Robust Contextual Reasoning in LLMscited by 1
  32. 2026GIF: Locally Sound Geometric Information Flow Control for LLMscited by 1
  33. 2026Agent Data Injection Attacks are Realistic Threats to AI Agentscited by 1
  34. 2026Prismata: Confining Cross-Site Prompt Injection in Web Agentscited by 1
  35. 2026IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requestscited by 1
  36. 2026Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Rankingcited by 1
  37. 2026Intellicise Wireless Networks Meet Agentic AI: A Security and Privacy Perspectivecited by 0
  38. 2026Policy Compliance of User Requests in Natural Language for AI Systemscited by 0

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
  2. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
  3. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
  4. 2026Evaluation of Prompt Injection Defenses in Large Language Models
  5. 2024An Early Categorization of Prompt Injection Attacks on Large Language Models
  6. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
  7. 2026When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems
  8. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise