mlsanityData to 7 Oct 2026

Prompt Injection attack against LLM-integrated Applications

arXiv 2306.05499 ↗First posted 8 Jun 2023cs.CR cs.AI cs.CLcited by 568

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. HouYi is compartmentalized into three crucial elements: a seamlessly-incorporated pre-constructed prompt, an injection prompt inducing context partition, and a malicious payload designed to fulfill the attack objectives. Leveraging HouYi, we unveil previously unknown and severe attack outcomes, such as unrestricted arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection. 10 vendors have validated our discoveries, including Notion, which has the potential to impact millions of users. Our investigation illuminates both the possible risks of prompt injection attacks and the possible tactics for mitigation.

Key facts. Cited by 568 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 32 of its 56 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

32 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 31

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  9. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  10. 2023Language Models can Solve Computer Taskscited by 371
  11. 2023Large Language Models as Tool Makerscited by 293
  12. 2023Role-Play with Large Language Modelscited by 290
19 more
  1. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  2. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  3. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  4. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  5. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  6. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  7. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  8. 2020Adversarial Training for Large Neural Language Modelscited by 109
  9. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  10. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  11. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  12. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 50
  13. 2022Contrastive Learning Reduces Hallucination in Conversationscited by 39
  14. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Modelscited by 33
  15. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  16. 2023ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedbackcited by 30
  17. 2021Interpreting the Robustness of Neural NLP Models to Textual Perturbationscited by 17
  18. 2021Get a Model! Model Hijacking Attack Against Machine Learning Modelscited by 13
  19. 2023Two-in-One: A Model Hijacking Attack Against Text Generation Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 137; 2025: 185; 2026: 210.

568 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  7. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  8. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  9. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  10. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  12. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
38 more
  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  5. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  6. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  7. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  8. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  9. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  10. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  11. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  12. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  13. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  14. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  15. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  16. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  17. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  18. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  19. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  20. 2025Securing AI Agents with Information-Flow Controlcited by 108
  21. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  22. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  23. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  24. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  25. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  26. 2024BadEdit: Backdooring large language models by model editingcited by 84
  27. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  28. 2023Effective Prompt Extraction from Language Modelscited by 83
  29. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  30. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  31. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  32. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  33. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  34. 2024Safeguarding Large Language Models: A Surveycited by 73
  35. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  36. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  37. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  38. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65

50 most cited of 568 All 568 (JSON)