mlsanityData to 7 Oct 2026

SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner

arXiv 2406.05498 ↗First posted 8 Jun 2024cs.CR cs.AIcited by 57

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreaking is an emerging adversarial attack that bypasses the safety alignment deployed in off-the-shelf large language models (LLMs) and has evolved into multiple categories: human-based, optimization-based, generation-based, and the recent indirect and multilingual jailbreaks. However, delivering a practical jailbreak defense is challenging because it needs to not only handle all the above jailbreak attacks but also incur negligible delays to user prompts, as well as be compatible with both open-source and closed-source LLMs. Inspired by how the traditional security concept of shadow stacks defends against memory overflow attacks, this paper introduces a generic LLM jailbreak defense framework called SelfDefend, which establishes a shadow LLM as a defense instance (in detection state) to concurrently protect the target LLM instance (in normal answering state) in the normal stack and collaborate with it for checkpoint-based access control. The effectiveness of SelfDefend builds upon our observation that existing LLMs can identify harmful prompts or intentions in user queries, which we empirically validate using mainstream GPT-3.5/4 models against major jailbreak attacks. To further improve the defense's robustness and minimize costs, we employ a data distillation approach to tune dedicated open-source defense models. When deployed to protect GPT-3.5/4, Claude, Llama-2-7b/13b, and Mistral, these models outperform seven state-of-the-art defenses and match the performance of GPT-4-based SelfDefend, with significantly lower extra delays. Further experiments show that the tuned models are robust to adaptive jailbreaks and prompt injections.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 81 of its 109 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

81 of 109 references.

Closest ancestors · 39

Same direction, newest first

  1. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs7 Jun
  2. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses3 Jun
  3. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models31 May
  4. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks30 May
  5. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing28 May
  6. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  7. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  8. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge8 Apr
  9. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  10. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  11. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models26 Mar
  12. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes1 Mar
27 more
  1. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  2. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  3. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper24 Feb
  4. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues14 Feb
  5. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  6. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuning9 Feb
  7. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  8. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  9. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Models30 Jan
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  11. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts15 Nov
  12. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  13. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  14. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  15. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  16. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  17. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  18. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  19. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  20. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  21. 2023Detecting Language Model Attacks with Perplexity27 Aug
  22. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  23. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  24. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  25. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  26. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  27. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
30 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  4. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  5. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  6. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  7. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  9. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  10. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  11. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  13. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  14. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  15. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  16. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  17. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  18. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  19. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  20. 2023GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysiscited by 65
  21. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  22. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  23. 2024PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generationcited by 36
  24. 2022CCTEST: Testing and Repairing Code Completion Systemscited by 35
  25. 2022State-specific protein-ligand complex structure prediction with a multi-scale deep generative modelcited by 35
  26. 2024An Empirical Evaluation of LLMs for Solving Offensive Security Challengescited by 33
  27. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Modelscited by 21
  28. 2023On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Studycited by 13
  29. 2018SCLib: A Practical and Lightweight Defense against Component Hijacking in Android Applicationscited by 3
  30. 2018Shining Light On Shadow Stackscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 28; 2026: 27.

57 papers.

Sort:

By area:

  1. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  2. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  3. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  4. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  5. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  6. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attackscited by 11
  7. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  8. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  9. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  10. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  11. 2025Distributional AGI Safetycited by 8
  12. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
38 more
  1. 2025Concept-Level Explainability for Auditing & Steering LLM Responsescited by 6
  2. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
  3. 2025Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganographycited by 6
  4. 2026JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attackscited by 6
  5. 2025Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attackscited by 5
  6. 2025STAC: When Innocent Tools Form Dangerous Chains for LLM Agentscited by 5
  7. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  8. 2025QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systemscited by 4
  9. 2026Hijacking Agent Memory: Stealthy Trojan Attacks Through Conversational Interactioncited by 4
  10. 2025STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Modelscited by 3
  11. 2025When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Outputcited by 3
  12. 2025Reasoning as a Resource: Optimizing Fast and Slow Thinking in Code Generation Modelscited by 3
  13. 2026ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplificationcited by 3
  14. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Modelscited by 3
  15. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  16. 2026Segment-Level Coherence for Robust Harmful Intent Probing in LLMscited by 3
  17. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  18. 2025ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Modelscited by 2
  19. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  20. 2026SafeDream: Safety World Model for Proactive Early Jailbreak Detectioncited by 2
  21. 2026RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetrycited by 2
  22. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  23. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  24. 2026SHIELD: An Auto-Healing Agentic Defense Framework for LLM Resource Exhaustion Attackscited by 1
  25. 2026Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defensecited by 1
  26. 2026From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrailscited by 1
  27. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  28. 2025Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Modelscited by 0
  29. 2025A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detectioncited by 0
  30. 2025Membership Inference Attacks on In-Context Examples in LLM-based Recommender Systemscited by 0
  31. 2025Digging Into the Internal: Causality-Based Analysis of LLM Function Callingcited by 0
  32. 2025Sanitize Your Responses: Mitigating Privacy Leakage in Large Language Modelscited by 0
  33. 2026Stop Testing Attacks, Start Diagnosing Defenses: The Four-Checkpoint Framework Reveals Where LLM Safety Breakscited by 0
  34. 2026SFCoT: Safer Chain-of-Thought via Active Safety Evaluation and Calibrationcited by 0
  35. 2026SelfGrader: LLM Jailbreak Detection via Anchored Token-Level Logitscited by 0
  36. 2026Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detectioncited by 0
  37. 2026Rewriting the Response Path: Silent Tampering and Provider-Signed Defense in BYOK LLM Agentscited by 0
  38. 2026Information Theoretic Adversarial Training of Large Language Modelscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  3. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
  4. 2025Proactive defense against LLM Jailbreak
  5. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  6. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  7. 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
  8. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers