mlsanityData to 14 Oct 2026

Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs

arXiv 2505.04806 ↗First posted 7 May 2025cs.CR cs.CLcited by 34

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) are increasingly integrated into consumer and enterprise applications. Despite their capabilities, they remain susceptible to adversarial attacks such as prompt injection and jailbreaks that override alignment safeguards. This paper provides a systematic investigation of jailbreak strategies against various state-of-the-art LLMs. We categorize over 1,400 adversarial prompts, analyze their success against GPT-4, Claude 2, Mistral 7B, and Vicuna, and examine their generalizability and construction logic. We further propose layered mitigation strategies and recommend a hybrid red-teaming and sandboxing approach for robust LLM security.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 42 of its 51 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

42 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems15 Apr
  2. 2024Playing Language Game with LLMs Leads to Jailbreaking16 Nov
  3. 2024Poisoned LangChain: Jailbreak LLMs by LangChain26 Jun
  4. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  6. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts7 Jun
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  12. 2022Toy Models of Superpositioncited by 971
23 more
  1. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  2. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  3. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  4. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  5. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  6. 2025Defeating Prompt Injections by Designcited by 297
  7. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  8. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  9. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 165
  10. 2024Stealing Part of a Production Language Modelcited by 159
  11. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 119
  12. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  13. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 54
  14. 2025PromptShield: Deployable Detection for Prompt Injection Attackscited by 50
  15. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  16. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 44
  17. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniquescited by 40
  18. 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 31
  19. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 30
  20. 2024Goal-guided Generative Prompt Injection Attack on Large Language Modelscited by 24
  21. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 16
  22. 2024Palisade -- Prompt Injection Detection Frameworkcited by 5
  23. 2024Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detectioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 18.

34 papers.

Sort:

By area:

  1. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  2. 2025POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimizationcited by 11
  3. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  4. 2025It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agentscited by 10
  5. 2025A Taxonomy of Prompt Defects in LLM Systemscited by 8
  6. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
  7. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Modelscited by 4
  8. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
  9. 2026An AI Agent Execution Environment to Safeguard User Datacited by 4
  10. 2025ANNIE: Be Careful of Your Robotscited by 3
  11. 2026Whispers of Wealth: Red-Teaming Google's Agent Payments Protocol via Prompt Injectioncited by 3
  12. 2026OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agentscited by 3
22 more
  1. 2025Jailbreaking in the Haystackcited by 2
  2. 2025Beyond Context: Large Language Models' Failure to Grasp Users' Intentcited by 1
  3. 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validitycited by 1
  4. 2026How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistencycited by 1
  5. 2026EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agentscited by 1
  6. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  7. 2025PrompTrend: Continuous Community-Driven Vulnerability Discovery and Assessment for Large Language Modelscited by 0
  8. 2025Data Security in Large Language Models: Risks, Defense, and Directionscited by 0
  9. 2025Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Datacited by 0
  10. 2025In AI Sweet Harmony: Sociopragmatic Guardrail Bypasses and Evaluation-Awareness in OpenAI gpt-oss-20bcited by 0
  11. 2025Conversational No-code, Multi-agentic Disease Module Identification and Drug Repurposing Prediction with ChatDRexcited by 0
  12. 2026Introducing the Generative Application Firewall (GAF)cited by 0
  13. 2026Security Assessment and Mitigation Strategies for Large Language Models: A Comprehensive Defensive Frameworkcited by 0
  14. 2026Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attackscited by 0
  15. 2026Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMscited by 0
  16. 2026kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrailcited by 0
  17. 2026Whitewashing Hate, Smearing Harmless Content: Annotator-Style Rebuttal Attacks on LLM-Based Moderationcited by 0
  18. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  19. 2026IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attackscited by 0
  20. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0
  21. 2026Does Moral Reasoning Training Help or Hurt? Red-Teaming RL-Trained Ethical Agents with Persona Attackscited by 0
  22. 2026Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
  2. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  3. 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
  4. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  5. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  6. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey
  7. 2026SoK: Robustness in Large Language Models against Jailbreak Attacks
  8. 2025RedTWIZ: Diverse LLM Red Teaming via Adaptive Attack Planning