mlsanityData to 7 Oct 2026

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

arXiv 2302.12173 ↗First posted 23 Feb 2023cs.CR cs.AI cs.CLcited by 1,339

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original instructions and employed controls. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We argue that LLM-Integrated Applications blur the line between data and instructions. We reveal new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We demonstrate our attacks' practical viability against both real-world systems, such as Bing's GPT-4 powered Chat and code-completion engines, and synthetic applications built on GPT-4. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing integration and reliance on LLMs, effective mitigations of these emerging threats are currently lacking. By raising awareness of these vulnerabilities and providing key insights into their implications, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users and systems from potential attacks.

Key facts. Cited by 1,339 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 86 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

26 of 86 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
14 more
  1. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  5. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  6. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  7. 2022Language Models as Agent Modelscited by 131
  8. 2022Evaluating Human-Language Model Interactioncited by 90
  9. 2021Bad Characters: Imperceptible NLP Attackscited by 69
  10. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  11. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  12. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  13. 2021Get a Model! Model Hijacking Attack Against Machine Learning Modelscited by 13
  14. 2021Understanding Emails and Drafting Responses -- An Approach Using GPT-3cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 54; 2024: 165; 2025: 296; 2026: 824.

1,339 papers.

Sort:

By area:

  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  7. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  11. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
38 more
  1. 2025Defeating Prompt Injections by Designcited by 256
  2. 2023Challenges and Applications of Large Language Modelscited by 251
  3. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  4. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  5. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  6. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  7. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  8. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  9. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  10. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  11. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  13. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  14. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  15. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  16. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  17. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  18. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  19. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  20. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  21. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  22. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  23. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  24. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  25. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  26. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  27. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  28. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  29. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  30. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  31. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  32. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systemscited by 118
  33. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  34. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  35. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  36. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  37. 2025Securing AI Agents with Information-Flow Controlcited by 108
  38. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108

50 most cited of 1,339 All 1,339 (JSON)