# Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs

arXiv: https://arxiv.org/abs/2505.04806 · HTML: https://mlsanity.com/paper/2505.04806/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 7 May 2025.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Automatic and Universal Prompt Injection Attacks against Large Language Models](https://mlsanity.com/paper/2403.04957/) (2024)
- [Attention Tracker: Detecting Prompt Injection Attacks in LLMs](https://mlsanity.com/paper/2411.00348/) (2024)
- [Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications](https://mlsanity.com/paper/2401.07612/) (2024)
- [PromptShield: Deployable Detection for Prompt Injection Attacks](https://mlsanity.com/paper/2501.15145/) (2025)

## Built on by (most cited first, 30 of 34)

- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization](https://mlsanity.com/paper/2508.19277/) (2025)
- [SoK: Trust-Authorization Mismatch in LLM Agent Interactions](https://mlsanity.com/paper/2512.06914/) (2025)
- [It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents](https://mlsanity.com/paper/2512.23128/) (2025)
- [A Taxonomy of Prompt Defects in LLM Systems](https://mlsanity.com/paper/2509.14404/) (2025)
- [Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety](https://mlsanity.com/paper/2508.03864/) (2025)
- [Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models](https://mlsanity.com/paper/2508.04196/) (2025)
- [Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models](https://mlsanity.com/paper/2602.01025/) (2026)
- [An AI Agent Execution Environment to Safeguard User Data](https://mlsanity.com/paper/2604.19657/) (2026)
- [ANNIE: Be Careful of Your Robots](https://mlsanity.com/paper/2509.03383/) (2025)
- [Whispers of Wealth: Red-Teaming Google's Agent Payments Protocol via Prompt Injection](https://mlsanity.com/paper/2601.22569/) (2026)
- [OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agents](https://mlsanity.com/paper/2605.08876/) (2026)
- [Jailbreaking in the Haystack](https://mlsanity.com/paper/2511.04707/) (2025)
- [Beyond Context: Large Language Models' Failure to Grasp Users' Intent](https://mlsanity.com/paper/2512.21110/) (2025)
- [Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity](https://mlsanity.com/paper/2601.06596/) (2026)
- [How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency](https://mlsanity.com/paper/2605.30096/) (2026)
- [EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agents](https://mlsanity.com/paper/2608.30429/) (2026)
- [Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking](https://mlsanity.com/paper/2507.08014/) (2025)
- [PrompTrend: Continuous Community-Driven Vulnerability Discovery and Assessment for Large Language Models](https://mlsanity.com/paper/2507.19185/) (2025)
- [Data Security in Large Language Models: Risks, Defense, and Directions](https://mlsanity.com/paper/2508.02312/) (2025)
- [Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Data](https://mlsanity.com/paper/2509.22850/) (2025)
- [In AI Sweet Harmony: Sociopragmatic Guardrail Bypasses and Evaluation-Awareness in OpenAI gpt-oss-20b](https://mlsanity.com/paper/2510.01259/) (2025)
- [Conversational No-code, Multi-agentic Disease Module Identification and Drug Repurposing Prediction with ChatDRex](https://mlsanity.com/paper/2511.21438/) (2025)
- [Introducing the Generative Application Firewall (GAF)](https://mlsanity.com/paper/2601.15824/) (2026)
- [Security Assessment and Mitigation Strategies for Large Language Models: A Comprehensive Defensive Framework](https://mlsanity.com/paper/2603.17123/) (2026)
- [Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks](https://mlsanity.com/paper/2604.01039/) (2026)
- [Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs](https://mlsanity.com/paper/2604.20945/) (2026)
- [kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail](https://mlsanity.com/paper/2607.02072/) (2026)
- [Whitewashing Hate, Smearing Harmless Content: Annotator-Style Rebuttal Attacks on LLM-Based Moderation](https://mlsanity.com/paper/2608.22230/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly integrated into consumer and enterprise applications. Despite their capabilities, they remain susceptible to adversarial attacks such as prompt injection and jailbreaks that override alignment safeguards. This paper provides a systematic investigation of jailbreak strategies against various state-of-the-art LLMs. We categorize over 1,400 adversarial prompts, analyze their success against GPT-4, Claude 2, Mistral 7B, and Vicuna, and examine their generalizability and construction logic. We further propose layered mitigation strategies and recommend a hybrid red-teaming and sandboxing approach for robust LLM security.