mlsanityData to 7 Oct 2026

Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents

arXiv 2503.15547 ↗First posted 17 Mar 2025cs.CR cs.AI cs.MAcited by 81

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large Language Models (LLMs) are combined with tools to create powerful LLM agents that provide a wide range of services. Unlike traditional software, LLM agent's behavior is determined at runtime by natural language prompts from either user or tool's data. This flexibility enables a new computing paradigm with unlimited capabilities and programmability, but also introduces new security risks, vulnerable to privilege escalation attacks. Moreover, user prompts are prone to be interpreted in an insecure way by LLM agents, creating non-deterministic behaviors that can be exploited by attackers. To address these security risks, we propose Prompt Flow Integrity (PFI), a system security-oriented solution to prevent privilege escalation in LLM agents. Analyzing the architectural characteristics of LLM agents, PFI features three mitigation techniques -- i.e., agent isolation, secure untrusted data processing, and privilege escalation guardrails. Our evaluation result shows that PFI effectively mitigates privilege escalation attacks while successfully preserving the utility of LLM agents.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 23 of its 59 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

23 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
  2. 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct

Other references · 21

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  10. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  11. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  12. 2024Stealing Part of a Production Language Modelcited by 152
9 more
  1. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  2. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  3. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 91
  4. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 29
  5. 2024Large Language Model Supply Chain: A Research Agendacited by 23
  6. 2024Permissive Information-Flow Analysis for Large Language Modelscited by 22
  7. 2015Android Permissions Remystified: A Field Study on Contextual Integritycited by 15
  8. 2023Stuck in the Permissions With You: Developer & End-User Perspectives on App Permissions & Their Privacy Ramificationscited by 2
  9. 2024SoK: Software Compartmentalizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 56.

81 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 256
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  3. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  4. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  5. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  6. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25
  7. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
  8. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  9. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  10. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  11. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  12. 2026From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skillscited by 15
38 more
  1. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  2. 2026CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attributioncited by 13
  3. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  4. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  5. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 12
  6. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 11
  7. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  8. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  9. 2026ChainCaps: Composition-Safe Tool-Using Agents via Monotonic Capability Attenuationcited by 11
  10. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  11. 2025SecInfer: Preventing Prompt Injection via Inference-time Scalingcited by 10
  12. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  13. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10
  14. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attackscited by 9
  15. 2025Preventing Prompt Injection with Type-Directed Privilege Separationcited by 8
  16. 2026Reframing LLM Agent Security as an Agent-Human Interaction Problemcited by 8
  17. 2025A Taxonomy of Prompt Defects in LLM Systemscited by 7
  18. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 7
  19. 2026Context-to-Execution Integrity for LLM Agentscited by 7
  20. 2025AI Awarenesscited by 6
  21. 2026When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agentcited by 6
  22. 2026Behavioral Integrity Verification for AI Agent Skillscited by 6
  23. 2026Assessing Automated Prompt Injection Attacks in Agentic Environmentscited by 5
  24. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  25. 2026Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agentscited by 5
  26. 2025Trusted AI Agents in the Cloudcited by 4
  27. 2026SkillScope: Toward Fine-Grained Least-Privilege Enforcement for Agent Skillscited by 4
  28. 2026AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agentscited by 4
  29. 2025LLM Agents Are the Antidote to Walled Gardenscited by 3
  30. 2026The LLMbda Calculus: AI Agents, Conversations, and Information Flowcited by 3
  31. 2026Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screeningcited by 3
  32. 2026Intent-Governed Tool Authorization for AI Agentscited by 3
  33. 2026Untrusted Content Masking for Web Agents with Security Guaranteescited by 3
  34. 2026OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skillscited by 3
  35. 2026Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memorycited by 3
  36. 2026CONTINUITY: Security-Context Contracts for Composable LLM Agent Controlscited by 3
  37. 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 2
  38. 2026AgenticCyOps: Securing Multi-Agentic AI Integration in Enterprise Cyber Operationscited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Prompt Control-Flow Integrity: A Priority-Aware Runtime Defense Against Prompt Injection in LLM Systems
  2. 2026AgentVisor: Defending LLM Agents Against Prompt Injection via Semantic Virtualization
  3. 2026Protecting Context and Prompts: Deterministic Security for Non-Deterministic AI
  4. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
  5. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  6. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
  7. 2026Securing LLM Agents Need Intent-to-Execution Integrity
  8. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents