# An Early Categorization of Prompt Injection Attacks on Large Language Models

arXiv: https://arxiv.org/abs/2402.00898 · HTML: https://mlsanity.com/paper/2402.00898/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://mlsanity.com/paper/2307.00691/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models](https://mlsanity.com/paper/2307.08487/) (2023)
- [Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection](https://mlsanity.com/paper/2308.10819/) (2023)

## Built on by (most cited first, 30 of 30)

- [Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs](https://mlsanity.com/paper/2505.04806/) (2025)
- [CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization](https://mlsanity.com/paper/2411.12768/) (2024)
- [Defending against Indirect Prompt Injection by Instruction Detection](https://mlsanity.com/paper/2505.06311/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [Prompt Injection as Role Confusion](https://mlsanity.com/paper/2603.12277/) (2026)
- [To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt](https://mlsanity.com/paper/2506.05739/) (2025)
- [Attack and defense techniques in large language models: A survey and new perspectives](https://mlsanity.com/paper/2505.00976/) (2025)
- [Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systems](https://mlsanity.com/paper/2603.00131/) (2026)
- [Towards More Realistic Extraction Attacks: An Adversarial Perspective](https://mlsanity.com/paper/2407.02596/) (2024)
- [Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions](https://mlsanity.com/paper/2603.03637/) (2026)
- [Gandalf the Red: Adaptive Security for LLMs](https://mlsanity.com/paper/2501.07927/) (2025)
- [Palisade -- Prompt Injection Detection Framework](https://mlsanity.com/paper/2410.21146/) (2024)
- [Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detection](https://mlsanity.com/paper/2503.21464/) (2025)
- [Too Big to Fool: Resisting Deception in Language Models](https://mlsanity.com/paper/2412.10558/) (2024)
- [You've Changed: Detecting Modification of Black-Box Large Language Models](https://mlsanity.com/paper/2504.12335/) (2025)
- [Countermind: A Multi-Layered Security Architecture for Large Language Models](https://mlsanity.com/paper/2510.11837/) (2025)
- [Proactive Hardening of LLM Defenses with HASTE](https://mlsanity.com/paper/2601.19051/) (2026)
- [An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Prompts](https://mlsanity.com/paper/2501.12521/) (2025)
- [Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models](https://mlsanity.com/paper/2505.00010/) (2025)
- [VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models](https://mlsanity.com/paper/2507.00052/) (2025)
- [CourtGuard: A Local, Multiagent Prompt Injection Classifier](https://mlsanity.com/paper/2510.19844/) (2025)
- [DAO-enabled decentralized physical AI: A new paradigm for human-machine collaboration](https://mlsanity.com/paper/2605.04522/) (2026)
- [A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection](https://mlsanity.com/paper/2508.07139/) (2025)
- [Design and Implementation of a Secure RAG-Enhanced AI Chatbot for Smart Tourism Customer Service: Defending Against Prompt Injection Attacks -- A Case Study of Hsinchu, Taiwan](https://mlsanity.com/paper/2509.21367/) (2025)
- [Goal-Driven Risk Assessment for LLM-Powered Systems: A Healthcare Case Study](https://mlsanity.com/paper/2603.03633/) (2026)
- [Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment](https://mlsanity.com/paper/2603.07460/) (2026)
- [Strengthening Polymorphic Prompt Assembling: Dynamic Separator Generation Against Emerging Prompt Injection Attacks](https://mlsanity.com/paper/2605.30534/) (2026)
- [The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems](https://mlsanity.com/paper/2607.19292/) (2026)
- [The Anatomy of a Prompt Injection: A Component Model for Structured Analysis](https://mlsanity.com/paper/2608.07808/) (2026)
- [BRANCH: Bypassing Multi-Scanner AI Guardrails](https://mlsanity.com/paper/2610.10742/) (2026)

## Abstract (arXiv)

Large language models and AI chatbots have been at the forefront of democratizing artificial intelligence. However, the releases of ChatGPT and other similar tools have been followed by growing concerns regarding the difficulty of controlling large language models and their outputs. Currently, we are witnessing a cat-and-mouse game where users attempt to misuse the models with a novel attack called prompt injections. In contrast, the developers attempt to discover the vulnerabilities and block the attacks simultaneously. In this paper, we provide an overview of these emergent threats and present a categorization of prompt injections, which can guide future research on prompt injections and act as a checklist of vulnerabilities in the development of LLM interfaces. Moreover, based on previous literature and our own empirical research, we discuss the implications of prompt injections to LLM end users, developers, and researchers.