mlsanityData to 7 Oct 2026

An Early Categorization of Prompt Injection Attacks on Large Language Models

arXiv 2402.00898 ↗First posted 31 Jan 2024cs.CR cs.CL cs.LGcited by 28

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large language models and AI chatbots have been at the forefront of democratizing artificial intelligence. However, the releases of ChatGPT and other similar tools have been followed by growing concerns regarding the difficulty of controlling large language models and their outputs. Currently, we are witnessing a cat-and-mouse game where users attempt to misuse the models with a novel attack called prompt injections. In contrast, the developers attempt to discover the vulnerabilities and block the attacks simultaneously. In this paper, we provide an overview of these emergent threats and present a categorization of prompt injections, which can guide future research on prompt injections and act as a checklist of vulnerabilities in the development of LLM interfaces. Moreover, based on previous literature and our own empirical research, we discuss the implications of prompt injections to LLM end users, developers, and researchers.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 20 of its 36 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

20 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection17 Aug
  2. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy3 Jul
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  10. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  11. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  12. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
4 more
  1. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  2. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 113
  3. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  4. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 9.

28 papers.

Sort:

By area:

  1. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  2. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  4. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  5. 2025To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Promptcited by 14
  6. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  7. 2026Prompt Injection as Role Confusioncited by 11
  8. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  9. 2026Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructionscited by 8
  10. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  11. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 7
  12. 2025Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detectioncited by 5
16 more
  1. 2024Palisade -- Prompt Injection Detection Frameworkcited by 3
  2. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
  3. 2025You've Changed: Detecting Modification of Black-Box Large Language Modelscited by 2
  4. 2025Countermind: A Multi-Layered Security Architecture for Large Language Modelscited by 2
  5. 2025An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Promptscited by 1
  6. 2025Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Modelscited by 1
  7. 2025VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Modelscited by 1
  8. 2025CourtGuard: A Local, Multiagent Prompt Injection Classifiercited by 1
  9. 2026DAO-enabled decentralized physical AI: A new paradigm for human-machine collaborationcited by 1
  10. 2025A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detectioncited by 0
  11. 2025Design and Implementation of a Secure RAG-Enhanced AI Chatbot for Smart Tourism Customer Service: Defending Against Prompt Injection Attacks -- A Case Study of Hsinchu, Taiwancited by 0
  12. 2026Goal-Driven Risk Assessment for LLM-Powered Systems: A Healthcare Case Studycited by 0
  13. 2026Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatmentcited by 0
  14. 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systemscited by 0
  15. 2026The Anatomy of a Prompt Injection: A Component Model for Structured Analysiscited by 0
  16. 2026BRANCH: Bypassing Multi-Scanner AI Guardrailscited by 0