mlsanityData to 7 Oct 2026

Defeating Prompt Injections by Design

arXiv 2503.18813 ↗First posted 24 Mar 2025cs.CR cs.AIcited by 256

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving 77% of tasks with provable security (compared to 84% with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.

Key facts. Cited by 256 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 31 of its 65 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

31 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents17 Mar
  2. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage13 Feb
  3. 2025Firewalls to Secure Dynamic LLM Agentic Networks3 Feb
  4. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  5. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agents8 May
  6. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems8 Mar

Other references · 25

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2022PAL: Program-aided Language Modelscited by 635
  10. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  12. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
13 more
  1. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  2. 2023Augmenting Language Models with Long-Term Memorycited by 186
  3. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 91
  4. 2023TaskWeaver: A Code-First Agent Frameworkcited by 85
  5. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  6. 2021Bad Characters: Imperceptible NLP Attackscited by 69
  7. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  8. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 36
  9. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33
  10. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 31
  11. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  12. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20
  13. 2024Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responsescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 57; 2026: 199.

256 papers.

Sort:

By area:

  1. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  2. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  3. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 114
  4. 2025Securing AI Agents with Information-Flow Controlcited by 108
  5. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  6. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  7. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 82
  8. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  9. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  10. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  11. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  12. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
38 more
  1. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  2. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  3. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  4. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  5. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  6. 2025Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injectionscited by 30
  7. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  8. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  9. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  10. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  11. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
  12. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 25
  13. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  14. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
  15. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  16. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challengecited by 20
  17. 2025Towards Automating Data Access Permissions in AI Agentscited by 20
  18. 2026SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agentscited by 20
  19. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  20. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
  21. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 17
  22. 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
  23. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  24. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 15
  25. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  26. 2026Learning to Inject: Automated Prompt Injection via Reinforcement Learningcited by 14
  27. 2026Ghost in the Agent: Redefining Information Flow Tracking for LLM Agentscited by 14
  28. 2026Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guaranteescited by 14
  29. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  30. 2026VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commitcited by 13
  31. 2026CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attributioncited by 13
  32. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  33. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13
  34. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  35. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 12
  36. 2025CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitizationcited by 12
  37. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  38. 2026Trojan Hippo: Weaponizing Agent Memory for Data Exfiltrationcited by 12

50 most cited of 256 All 256 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentVisor: Defending LLM Agents Against Prompt Injection via Semantic Virtualization
  2. 2024Whispers in the Machine: Confidentiality in Agentic Systems
  3. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  4. 2026CaMeLoT: CaMeL orchestrated with Temporal logic for static verification and liveness
  5. 2026SPA: Securing Persistent LLM Agents Across Queries with Plan-First Information-Flow Control
  6. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  7. 2026Protecting Context and Prompts: Deterministic Security for Non-Deterministic AI
  8. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows