mlsanityData to 7 Oct 2026

Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models

arXiv 2312.14197 ↗First posted 21 Dec 2023cs.CL cs.AIcited by 237

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

The integration of large language models with external content has enabled applications such as Microsoft Copilot but also introduced vulnerabilities to indirect prompt injection attacks. In these attacks, malicious instructions embedded within external content can manipulate LLM outputs, causing deviations from user expectations. To address this critical yet under-explored issue, we introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Using BIPIA, we evaluate existing LLMs and find them universally vulnerable. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. Based on these findings, we propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings. Extensive experiments demonstrate that our black-box defense provides substantial mitigation, while our white-box defense reduces the attack success rate to near-zero levels, all while preserving the output quality of LLMs. We hope this work inspires further research into securing LLM applications and fostering their safe and reliable use.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 52 of its 78 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

52 of 78 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
37 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023Mistral 7Bcited by 3,065
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  13. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  14. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  15. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  16. 2023LIMA: Less Is More for Alignmentcited by 1,080
  17. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  18. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  19. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  20. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  21. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  22. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  23. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  24. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  25. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  26. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  27. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  28. 2023Augmented Language Models: a Surveycited by 398
  29. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  30. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  31. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  32. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  33. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  34. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  35. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  36. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  37. 2023LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Pluginscited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 72; 2026: 127.

237 papers.

Sort:

By area:

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  5. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  6. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  7. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  8. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  9. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  11. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  2. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  3. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  4. 2025Securing AI Agents with Information-Flow Controlcited by 108
  5. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  6. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  7. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 81
  8. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  9. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  10. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  11. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  12. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  13. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  14. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  15. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  16. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 50
  17. 2025MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocolscited by 47
  18. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  19. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  20. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  21. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  22. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  23. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  24. 2025A Closer Look at System Prompt Robustnesscited by 31
  25. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  26. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
  27. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  28. 2025LLM Agents Should Employ Security Principlescited by 28
  29. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  30. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  31. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  32. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  33. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 25
  34. 2024Embedding-based classifiers can detect prompt injection attackscited by 24
  35. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  36. 2024Ranking Manipulation for Conversational Search Enginescited by 23
  37. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  38. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 22

50 most cited of 237 All 237 (JSON)