mlsanityData to 14 Oct 2026

A Critical Evaluation of Defenses against Prompt Injection Attacks

arXiv 2505.18333 ↗First posted 23 May 2025cs.CR cs.AIcited by 36

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are vulnerable to prompt injection attacks, and several defenses have recently been proposed, often claiming to mitigate these attacks successfully. However, we argue that existing studies lack a principled approach to evaluating these defenses. In this paper, we argue the need to assess defenses across two critical dimensions: (1) effectiveness, measured against both existing and adaptive prompt injection attacks involving diverse target and injected prompts, and (2) general-purpose utility, ensuring that the defense does not compromise the foundational capabilities of the LLM. Our critical evaluation reveals that prior studies have not followed such a comprehensive evaluation methodology. When assessed using this principled approach, we show that existing defenses are not as successful as previously reported. This work provides a foundation for evaluating future defenses and guiding their development. Our code and data are available at: https://github.com/PIEval123/PIEval.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 41 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

26 of 41 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Prompt Injection Attack to Tool Selection in LLM Agents28 Apr
  2. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks15 Apr
  3. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMs1 Nov
  4. 2024Embedding-based classifiers can detect prompt injection attacks29 Oct
  5. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy9 Oct
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  7. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge26 Mar
  8. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models7 Mar
  9. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks6 Mar
  10. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  11. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  12. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defenses19 Oct
1 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 13

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  10. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  11. 2023Detecting Language Model Attacks with Perplexitycited by 431
  12. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
1 more
  1. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 27.

36 papers.

Sort:

By area:

  1. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 108
  2. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 33
  3. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  4. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  5. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  6. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 24
  7. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 18
  8. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  9. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  10. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  11. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 10
  12. 2025SecInfer: Preventing Prompt Injection via Inference-time Scalingcited by 10
24 more
  1. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 9
  2. 2025Fingerprinting LLMs via Prompt Injectioncited by 8
  3. 2026AgentWatcher: A Rule-based Prompt Injection Monitorcited by 6
  4. 2026AlignSentinel: Alignment-Aware Detection of Prompt Injection Attackscited by 5
  5. 2026Agent Security is a Systems Problemcited by 5
  6. 2026Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screeningcited by 5
  7. 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agentscited by 3
  8. 2026PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysiscited by 3
  9. 2026Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMscited by 2
  10. 2026LocalAlign: Enabling Generalizable Prompt Injection Defense via Generation of Near-Target Adversarial Examples for Alignment Trainingcited by 1
  11. 2026Prismata: Confining Cross-Site Prompt Injection in Web Agentscited by 1
  12. 2026FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruptioncited by 0
  13. 2026CleanBase: Detecting Malicious Documents in RAG Knowledge Databasescited by 0
  14. 2026On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gapcited by 0
  15. 2026Investigating Detection and Obfuscation of Prompt Injection Attacks Against Software Reverse Engineering AI Agentscited by 0
  16. 2026Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agentscited by 0
  17. 2026Rethinking Agent Security as a Networking Problemcited by 0
  18. 2026Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matchingcited by 0
  19. 2026Rethinking Indirect Prompt Injection as a Test-Time Search Problemcited by 0
  20. 2026Detecting and Localizing Segment-Level Poisoning in Multi-Source LLM-Agent Inputscited by 0
  21. 2026A Proposal for an Agentic AI Architecture to Support Multi-Domain Decision-Making in the Brazilian Armed Forcescited by 0
  22. 2026Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injectioncited by 0
  23. 2026ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agentscited by 0
  24. 2026TRACER: Trace-Regularized Training Against Indirect Prompt Injectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AttackEval: A Systematic Empirical Study of Prompt Injection Attack Effectiveness Against Large Language Models
  2. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniques
  3. 2024Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures
  4. 2025Attack and defense techniques in large language models: A survey and new perspectives
  5. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  6. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  7. 2024An Early Categorization of Prompt Injection Attacks on Large Language Models
  8. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models