mlsanityData to 7 Oct 2026

Jailbreak Attacks and Defenses Against Large Language Models: A Survey

arXiv 2407.04295 ↗First posted 5 Jul 2024cs.CR cs.AI cs.CLcited by 215

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) have performed exceptionally in various text-generative tasks, including question answering, translation, code completion, etc. However, the over-assistance of LLMs has raised the challenge of "jailbreaking", which induces the model to generate malicious responses against the usage policy and society by designing adversarial prompts. With the emergence of jailbreak attack methods exploiting different vulnerabilities in LLMs, the corresponding safety alignment measures are also evolving. In this paper, we propose a comprehensive and detailed taxonomy of jailbreak attack and defense methods. For instance, the attack methods are divided into black-box and white-box attacks based on the transparency of the target model. Meanwhile, we classify defense methods into prompt-level and model-level defenses. Additionally, we further subdivide these attack and defense methods into distinct sub-classes and present a coherent diagram illustrating their relationships. We also conduct an investigation into the current evaluation methods and compare them from different perspectives. Our findings aim to inspire future research and practical implementations in safeguarding LLMs against adversarial attacks. Above all, although jailbreak remains a significant concern within the community, we believe that our work enhances the understanding of this domain and provides a foundation for developing more secure LLMs.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 109 of its 122 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

109 of 122 references.

Closest ancestors · 56

Same direction, newest first

  1. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses3 Jun
  2. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusal25 Apr
  3. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  5. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models18 Mar
  6. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  7. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes1 Mar
  8. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction28 Feb
  9. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models26 Feb
  10. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  11. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  12. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
38 more
  1. 2024Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement23 Feb
  2. 2024Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs21 Feb
  3. 2024Coercing LLMs to do and reveal (almost) anything21 Feb
  4. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers16 Feb
  5. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  6. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  7. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues14 Feb
  8. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  9. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  10. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning13 Feb
  11. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  12. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models5 Feb
  13. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  14. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  15. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Models30 Jan
  16. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  17. 2024Intention Analysis Makes LLMs A Good Jailbreak Defender12 Jan
  18. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  19. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  20. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  21. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  22. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  23. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  24. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  25. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks16 Oct
  26. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  27. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  28. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  29. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  30. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  31. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  32. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  33. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  34. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  35. 2023FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models11 Sep
  36. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  37. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  38. 2023Detecting Language Model Attacks with Perplexity27 Aug

50 newest of 56 All 56 (JSON)

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
38 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2023Large Language Modelscited by 774
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  10. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  11. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  12. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  13. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  14. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  15. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  16. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  17. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  18. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  19. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  20. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  21. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  22. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  23. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacycited by 137
  24. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  25. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  26. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  27. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  28. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  29. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  30. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  31. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
  32. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  33. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  34. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33
  35. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 26
  36. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 26
  37. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMscited by 23
  38. 2023Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principlescited by 17

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 117; 2026: 72.

215 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  3. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  4. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  5. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  6. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  7. 2025Safety in Large Reasoning Models: A Surveycited by 58
  8. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  9. 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
  10. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 39
  11. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  12. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
38 more
  1. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  2. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  3. 2025Enforcing Temporal Constraints for LLM Agentscited by 31
  4. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  5. 2026Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threatscited by 25
  6. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  8. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 24
  9. 2025Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judgecited by 23
  10. 2025Thought Manipulation: External Thought Can Be Efficient for Large Reasoning Modelscited by 23
  11. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  12. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  13. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  14. 2026SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agentscited by 20
  15. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 19
  16. 2025T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attackscited by 18
  17. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  18. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 17
  19. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  20. 2025Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Searchcited by 15
  21. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 15
  22. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  23. 2025To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Promptcited by 14
  24. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  25. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  26. 2026From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaborationcited by 13
  27. 2024Quantized Delta Weight Is Safety Keepercited by 12
  28. 2025Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviorscited by 12
  29. 2025LLM-Enabled In-Context Learning for Data Collection Scheduling in UAV-assisted Sensor Networkscited by 12
  30. 2025Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systemscited by 12
  31. 2025HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessmentcited by 12
  32. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
  33. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  34. 2025POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimizationcited by 11
  35. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  36. 2025Are Your Agents Upward Deceivers?cited by 11
  37. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroringcited by 10
  38. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMscited by 10

50 most cited of 215 All 215 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  2. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  3. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  4. 2024textitMMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models
  5. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  6. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  7. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
  8. 2025Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?