mlsanityData to 14 Oct 2026

Jailbreaking and Mitigation of Vulnerabilities in Large Language Models

arXiv 2410.15236 ↗First posted 20 Oct 2024cs.CR cs.AI cs.LGcited by 34

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) have transformed artificial intelligence by advancing natural language understanding and generation, enabling applications across fields beyond healthcare, software engineering, and conversational systems. Despite these advancements in the past few years, LLMs have shown considerable vulnerabilities, particularly to prompt injection and jailbreaking attacks. This review analyzes the state of research on these vulnerabilities and presents available defense strategies. We roughly categorize attack approaches into prompt-based, model-based, multimodal, and multilingual, covering techniques such as adversarial prompting, backdoor injections, and cross-modality exploits. We also review various defense mechanisms, including prompt filtering, transformation, alignment techniques, multi-agent defenses, and self-regulation, evaluating their strengths and shortcomings. We also discuss key metrics and benchmarks used to assess LLM safety and robustness, noting challenges like the quantification of attack success in interactive contexts and biases in existing datasets. Identifying current research gaps, we suggest future directions for resilient alignment strategies, advanced defenses against evolving attacks, automation of jailbreak detection, and consideration of ethical and societal impacts. This review emphasizes the need for continued research and cooperation within the AI community to enhance LLM security and ensure their safe deployment.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 83 of its 98 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

83 of 98 references.

Closest ancestors · 45

Same direction, newest first

  1. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response22 May
  2. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetition13 May
  3. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge8 Apr
  4. 2024Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injection7 Apr
  5. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  7. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  8. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  9. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models26 Mar
  10. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  11. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction28 Feb
  12. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue27 Feb
33 more
  1. 2024Defending LLMs against Jailbreaking Attacks via Backtranslation26 Feb
  2. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper24 Feb
  3. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  4. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuning9 Feb
  5. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  6. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  7. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Models30 Jan
  8. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning19 Jan
  9. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective12 Jan
  10. 2024Intention Analysis Makes LLMs A Good Jailbreak Defender12 Jan
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  12. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts15 Nov
  13. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  14. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  15. 2023Self-Guard: Empower the LLM to Safeguard Itself24 Oct
  16. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  17. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  18. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  19. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  20. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  21. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  22. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  23. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  24. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  25. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  26. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  27. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  28. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  29. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models17 Jul
  30. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  31. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  32. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks24 May
  33. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  10. 2022Red Teaming Language Models with Language Modelscited by 1,026
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
26 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  4. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  8. 2023AI Alignment: A Comprehensive Surveycited by 263
  9. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  10. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  11. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  12. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  13. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  14. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  15. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 136
  16. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  17. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  18. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 89
  19. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  20. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
  21. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 67
  22. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 49
  23. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Modelscited by 42
  24. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  25. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
  26. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 16; 2026: 12.

34 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 99
  4. 2025Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial Statescited by 9
  5. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 8
  6. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Modelscited by 8
  7. 2025Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganographycited by 8
  8. 2025The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Searchcited by 7
  9. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  10. 2025Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigationcited by 5
  11. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  12. 2024The Cat and Mouse Game: The Ongoing Arms Race Between Diffusion Models and Detection Methodscited by 4
22 more
  1. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  2. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensingcited by 3
  3. 2024From Noise to Nuance: Advances in Deep Generative Image Modelscited by 3
  4. 2025XBreaking: Understanding how LLMs security alignment can be brokencited by 3
  5. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 3
  6. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  7. 2025Model Risk Management for Generative AI In Financial Institutionscited by 2
  8. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputscited by 2
  9. 2026BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languagescited by 2
  10. 2026An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Modelscited by 2
  11. 2026When Behavioral Safety Evaluation Fails: A Representation-Level Perspectivecited by 2
  12. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  13. 2025ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attackcited by 1
  14. 2025Sentra-Guard: A Real-Time Multilingual Defense Against Adversarial LLM Promptscited by 1
  15. 2026Auditing Cascading Risks in Multi-Agent Systems via Semantic-Geometric Co-evolutioncited by 1
  16. 2026Multilingual jailbreaking of LLMs using low-resource languagescited by 1
  17. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 1
  18. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Modelscited by 0
  19. 2026Fully Homomorphic Encryption on Llama 3 model for privacy preserving LLM inferencecited by 0
  20. 2026uGen: An Agentic Framework for Generating Microarchitectural Attack PoCscited by 0
  21. 2026Evolving and Detecting Multi-Turn Deception using Geometric Signaturescited by 0
  22. 2026Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  2. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey
  3. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  4. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models
  5. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  6. 2025Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
  7. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  8. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models