# A Critical Evaluation of Defenses against Prompt Injection Attacks

arXiv: https://arxiv.org/abs/2505.18333 · HTML: https://mlsanity.com/paper/2505.18333/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 26 of 26)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [Prompt Injection Attack to Tool Selection in LLM Agents](https://mlsanity.com/paper/2504.19793/) (2025)
- [Optimization-based Prompt Injection Attack to LLM-as-a-Judge](https://mlsanity.com/paper/2403.17710/) (2024)
- [DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2504.11358/) (2025)
- [PLeak: Prompt Leaking Attacks against Large Language Model Applications](https://mlsanity.com/paper/2405.06823/) (2024)
- [Automatic and Universal Prompt Injection Attacks against Large Language Models](https://mlsanity.com/paper/2403.04957/) (2024)
- [Attention Tracker: Detecting Prompt Injection Attacks in LLMs](https://mlsanity.com/paper/2411.00348/) (2024)
- [Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks](https://mlsanity.com/paper/2403.03792/) (2024)
- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [Embedding-based classifiers can detect prompt injection attacks](https://mlsanity.com/paper/2410.22284/) (2024)

## Built on by (most cited first, 30 of 36)

- [PromptArmor: Simple yet Effective Prompt Injection Defenses](https://mlsanity.com/paper/2507.15219/) (2025)
- [Formal Policy Enforcement for Real-World Agentic Systems](https://mlsanity.com/paper/2602.16708/) (2026)
- [PromptLocate: Localizing Prompt Injection Attacks](https://mlsanity.com/paper/2510.12252/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools](https://mlsanity.com/paper/2508.02110/) (2025)
- [Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools](https://mlsanity.com/paper/2509.21011/) (2025)
- [ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data](https://mlsanity.com/paper/2512.09321/) (2025)
- [A Framework for Formalizing LLM Agent Security](https://mlsanity.com/paper/2603.19469/) (2026)
- [PIArena: A Platform for Prompt Injection Evaluation](https://mlsanity.com/paper/2604.08499/) (2026)
- [Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks](https://mlsanity.com/paper/2507.02735/) (2025)
- [SecInfer: Preventing Prompt Injection via Inference-time Scaling](https://mlsanity.com/paper/2509.24967/) (2025)
- [The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism](https://mlsanity.com/paper/2601.09625/) (2026)
- [Fingerprinting LLMs via Prompt Injection](https://mlsanity.com/paper/2509.25448/) (2025)
- [AgentWatcher: A Rule-based Prompt Injection Monitor](https://mlsanity.com/paper/2604.01194/) (2026)
- [AlignSentinel: Alignment-Aware Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2602.13597/) (2026)
- [Agent Security is a Systems Problem](https://mlsanity.com/paper/2605.18991/) (2026)
- [Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening](https://mlsanity.com/paper/2605.28999/) (2026)
- [Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents](https://mlsanity.com/paper/2606.26479/) (2026)
- [PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis](https://mlsanity.com/paper/2607.12624/) (2026)
- [Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs](https://mlsanity.com/paper/2601.12359/) (2026)
- [LocalAlign: Enabling Generalizable Prompt Injection Defense via Generation of Near-Target Adversarial Examples for Alignment Training](https://mlsanity.com/paper/2605.01462/) (2026)
- [Prismata: Confining Cross-Site Prompt Injection in Web Agents](https://mlsanity.com/paper/2607.08147/) (2026)
- [FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruption](https://mlsanity.com/paper/2604.28157/) (2026)
- [CleanBase: Detecting Malicious Documents in RAG Knowledge Databases](https://mlsanity.com/paper/2605.00460/) (2026)
- [On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gap](https://mlsanity.com/paper/2605.19159/) (2026)
- [Investigating Detection and Obfuscation of Prompt Injection Attacks Against Software Reverse Engineering AI Agents](https://mlsanity.com/paper/2605.30677/) (2026)
- [Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agents](https://mlsanity.com/paper/2607.18826/) (2026)
- [Rethinking Agent Security as a Networking Problem](https://mlsanity.com/paper/2608.12172/) (2026)
- [Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching](https://mlsanity.com/paper/2609.02035/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are vulnerable to prompt injection attacks, and several defenses have recently been proposed, often claiming to mitigate these attacks successfully. However, we argue that existing studies lack a principled approach to evaluating these defenses. In this paper, we argue the need to assess defenses across two critical dimensions: (1) effectiveness, measured against both existing and adaptive prompt injection attacks involving diverse target and injected prompts, and (2) general-purpose utility, ensuring that the defense does not compromise the foundational capabilities of the LLM. Our critical evaluation reveals that prior studies have not followed such a comprehensive evaluation methodology. When assessed using this principled approach, we show that existing defenses are not as successful as previously reported. This work provides a foundation for evaluating future defenses and guiding their development. Our code and data are available at: https://github.com/PIEval123/PIEval.