mlsanityData to 7 Oct 2026

Formalizing and Benchmarking Prompt Injection Attacks and Defenses

arXiv 2310.12815 ↗First posted 19 Oct 2023cs.CR cs.AI cs.CLcited by 328

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

A prompt injection attack aims to inject malicious instruction/data into the input of an LLM-Integrated Application such that it produces results as an attacker desires. Existing works are limited to case studies. As a result, the literature lacks a systematic understanding of prompt injection attacks and their defenses. We aim to bridge the gap in this work. In particular, we propose a framework to formalize prompt injection attacks. Existing attacks are special cases in our framework. Moreover, based on our framework, we design a new attack by combining existing ones. Using our framework, we conduct a systematic evaluation on 5 prompt injection attacks and 10 defenses with 10 LLMs and 7 tasks. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses. To facilitate research on this topic, we make our platform public at https://github.com/liu00222/Open-Prompt-Injection.

Key facts. Cited by 328 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 24 of its 59 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

24 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  10. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2023Detecting Language Model Attacks with Perplexitycited by 411
10 more
  1. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  2. 2022UL2: Unifying Language Learning Paradigmscited by 292
  3. 2023Poisoning Language Models During Instruction Tuningcited by 254
  4. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  5. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  6. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  7. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  8. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  9. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  10. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 29; 2025: 107; 2026: 190.

328 papers.

Sort:

By area:

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  3. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  4. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  5. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  6. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  7. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  8. 2025Securing AI Agents with Information-Flow Controlcited by 108
  9. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  10. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  11. 2026Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scalecited by 94
  12. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
38 more
  1. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  2. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 64
  3. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  4. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  5. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
  6. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  7. 2026"Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wildcited by 46
  8. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
  9. 2026Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injectionscited by 44
  10. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  11. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  12. 2024Voice Jailbreak Attacks Against GPT-4ocited by 39
  13. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  14. 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 35
  15. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  16. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  17. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  18. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  19. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  20. 2025LLM Agents Should Employ Security Principlescited by 28
  21. 2025Free-MAD: Consensus-Free Multi-Agent Debatecited by 27
  22. 2025Security Concerns for Large Language Models: A Surveycited by 26
  23. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  24. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  25. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  26. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 25
  27. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  28. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
  29. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  30. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  31. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  32. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20
  33. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20
  34. 2026SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agentscited by 20
  35. 2025Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Modelscited by 19
  36. 2025Secure Retrieval-Augmented Generation against Poisoning Attackscited by 19
  37. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 18
  38. 2025ETDI: Mitigating Tool Squatting and Rug Pull Attacks in Model Context Protocol (MCP) by using OAuth-Enhanced Tool Definitions and Policy-Based Access Controlcited by 18

50 most cited of 328 All 328 (JSON)