mlsanityData to 14 Oct 2026

System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective

arXiv 2409.19091 ↗First posted 27 Sep 2024cs.CRcited by 96

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Model-based systems (LLM systems) are information and query processing systems that use LLMs to plan operations from natural-language prompts and feed the output of each successive step into the LLM to plan the next. This structure results in powerful tools that can process complex information from diverse sources but raises critical security concerns. Malicious information from any source may be processed by the LLM and can compromise the query processing, resulting in nearly arbitrary misbehavior. To tackle this problem, we present a system-level defense based on the principles of information flow control that we call an f-secure LLM system. An f-secure LLM system disaggregates the components of an LLM system into a context-aware pipeline with dynamically generated structured executable plans, and a security monitor filters out untrusted input into the planning process. This structure prevents compromise while maximizing flexibility. We provide formal models for both existing LLM systems and our f-secure LLM system, allowing analysis of critical security guarantees. We further evaluate case studies and benchmarks showing that f-secure LLM systems provide robust security while preserving functionality and efficiency. Our code is released at https://github.com/fzwark/Secure_LLM_System.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 24 of its 55 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

24 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  2. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications15 Jan
  3. 2024A Novel Evaluation Framework for Assessing Resilience Against Prompt Injection Attacks in Large Language Models2 Jan
  4. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning29 Dec
  5. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  6. 2023Maatphor: Automated Variant Analysis for Prompt Injection Attacks12 Dec
  7. 2023Assessing Prompt Injection Risks in 200+ Custom GPTs20 Nov
  8. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  9. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?3 Aug
  10. 2023Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection15 Jun
  11. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 12

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  6. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  7. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systemscited by 130
  8. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 86
  9. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 77
  10. 2023LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Pluginscited by 51
  11. 2017Nonmalleable Information Flow: Technical Reportcited by 4
  12. 2021Compositional Security for Reentrant Applicationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 37; 2026: 58.

96 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 297
  2. 2025Progent: Securing AI Agents with Privilege Controlcited by 141
  3. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  4. 2025Securing AI Agents with Information-Flow Controlcited by 121
  5. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 108
  6. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 87
  7. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 77
  8. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  9. 2025Contextual Agent Security: A Policy for Every Purposecited by 59
  10. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
  11. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 53
  12. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 46
38 more
  1. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  2. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 38
  3. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 36
  4. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 32
  5. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  6. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  7. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  8. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  9. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 24
  10. 2024Permissive Information-Flow Analysis for Large Language Modelscited by 22
  11. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  12. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  13. 2025Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?cited by 18
  14. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  15. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 17
  16. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 16
  17. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  18. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 15
  19. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 15
  20. 2025Systems Security Foundations for Agentic Computingcited by 13
  21. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 12
  22. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  23. 2025The Hidden Dangers of Browsing AI Agentscited by 11
  24. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Spacecited by 11
  25. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  26. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 11
  27. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  28. 2025SecInfer: Preventing Prompt Injection via Inference-time Scalingcited by 10
  29. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  30. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  31. 2026Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agentscited by 10
  32. 2025Preventing Prompt Injection with Type-Directed Privilege Separationcited by 9
  33. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 9
  34. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  35. 2026Reframing LLM Agent Security as an Agent-Human Interaction Problemcited by 9
  36. 2025Maris: A Formally Verifiable Privacy Policy Enforcement Paradigm for Multi-Agent Collaboration Systemscited by 8
  37. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 7
  38. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 7

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems
  2. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  3. 2026Prompt Control-Flow Integrity: A Priority-Aware Runtime Defense Against Prompt Injection in LLM Systems
  4. 2025Security Concerns for Large Language Models: A Survey
  5. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  6. 2026Evaluation of Prompt Injection Defenses in Large Language Models
  7. 2024StruQ: Defending Against Prompt Injection with Structured Queries
  8. 2026Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning