mlsanityData to 14 Oct 2026

A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models

arXiv 2402.13457 ↗First posted 21 Feb 2024cs.CR cs.AIcited by 106

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMS) have increasingly become central to generating content with potential societal impacts. Notably, these models have demonstrated capabilities for generating content that could be deemed harmful. To mitigate these risks, researchers have adopted safety training techniques to align model outputs with societal values to curb the generation of malicious content. However, the phenomenon of "jailbreaking", where carefully crafted prompts elicit harmful responses from models, persists as a significant challenge. This research conducts a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. We meticulously investigate nine attack techniques and seven defense techniques applied across three distinct language models: Vicuna, LLama, and GPT-3.5 Turbo. We aim to evaluate the effectiveness of these attack and defense techniques. Our findings reveal that existing white-box attacks underperform compared to universal techniques and that including special tokens in the input significantly affects the likelihood of successful attacks. This research highlights the need to concentrate on the security facets of LLMs. Additionally, we contribute to the field by releasing our datasets and testing framework, aiming to foster further research into LLM security. We believe these contributions will facilitate the exploration of security measures within this domain.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 30 of its 56 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

30 of 56 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information20 Nov
  2. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking16 Nov
  3. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts15 Nov
  4. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  5. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  9. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  11. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  12. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
3 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  3. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  5. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  7. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  8. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  9. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  10. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  11. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  12. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 45
3 more
  1. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Frameworkcited by 30
  2. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  3. 2023Why do universal adversarial attacks work on large language models?: Geometry might be the answercited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 59; 2026: 25.

106 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  3. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
  4. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  5. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 64
  6. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 56
  7. 2025Large Language Models Pass the Turing Testcited by 48
  8. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 32
  9. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  10. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  11. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  12. 2025ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinkingcited by 23
38 more
  1. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 21
  2. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 20
  3. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 18
  4. 2025Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Searchcited by 16
  5. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 15
  6. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  7. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14
  8. 2024Efficient Detection of Toxic Prompts in Large Language Modelscited by 13
  9. 2024Quantized Delta Weight Is Safety Keepercited by 12
  10. 2025On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignmentcited by 12
  11. 2024Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injectioncited by 9
  12. 2025Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMscited by 9
  13. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 9
  14. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  15. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attackscited by 8
  16. 2024OpenThaiGPT 1.5: A Thai-Centric Open Source Large Language Modelcited by 8
  17. 2025"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Modelscited by 8
  18. 2025Build Agent Advocates, Not Platform Agentscited by 8
  19. 2024Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Modelscited by 7
  20. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 7
  21. 2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 7
  22. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  23. 2025Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attackscited by 6
  24. 2025Exposing the Ghost in the Transformer: Abnormal Detection for Large Language Models via Hidden State Forensicscited by 6
  25. 2025Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generationcited by 6
  26. 2025ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusalcited by 6
  27. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  28. 2025RAI: Flexible Agent Framework for Embodied AIcited by 5
  29. 2025Adversarial Suffix Filtering: a Defense Pipeline for LLMscited by 5
  30. 2025Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narrativescited by 5
  31. 2025Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigmcited by 5
  32. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  33. 2025Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attackscited by 5
  34. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Queryingcited by 5
  35. 2024Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Modelscited by 4
  36. 2024SQL Injection Jailbreak: A Structural Disaster of Large Language Modelscited by 4
  37. 2024On the Ethical Considerations of Generative Agentscited by 4
  38. 2025Beyond Benchmarks: On The False Promise of AI Regulationcited by 4

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  3. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  4. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense
  5. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Models
  6. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  7. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
  8. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs