mlsanityData to 7 Oct 2026

Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

arXiv 2412.17034 ↗First posted 22 Dec 2024cs.CLcited by 23

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreaking in Large Language Models (LLMs) is a major security concern as it can deceive LLMs to generate harmful text. Yet, there is still insufficient understanding of how jailbreaking works, which makes it hard to develop effective defense strategies. We aim to shed more light into this issue: we conduct a detailed large-scale analysis of seven different jailbreak methods and find that these disagreements stem from insufficient observation samples. In particular, we introduce textitsafety boundary, and we find that jailbreaks shift harmful activations outside that safety boundary, where LLMs are less sensitive to harmful information. We also find that the low and the middle layers are critical in such shifts, while deeper layers have less impact. Leveraging on these insights, we propose a novel defense called Activation Boundary Defense (ABD), which adaptively constrains the activations within the safety boundary. We further use Bayesian optimization to selectively apply the defense method to the low and the middle layers. Our experiments on several benchmarks show that ABD achieves an average DSR of over 98% against various forms of jailbreak attacks, with less than 2% impact on the model's general capabilities.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 24 of its 58 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

24 of 58 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit17 Nov
  2. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles8 Aug
  3. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis16 Jun
  4. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States9 Jun
  5. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing28 May
  6. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis21 Feb
  7. 2024Defending Jailbreak Prompts via In-Context Adversarial Game20 Feb
  8. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  9. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  10. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  11. 2024Intention Analysis Makes LLMs A Good Jailbreak Defender12 Jan
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
5 more
  1. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  4. 2023Detecting Language Model Attacks with Perplexity27 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 7

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  6. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  7. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  2. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  3. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  4. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  5. 2026Analysing the Safety Pitfalls of Steering Vectorscited by 9
  6. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  7. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbationscited by 4
  8. 2026The Cylindrical Representation Hypothesis for Language Model Steeringcited by 3
  9. 2026MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attackscited by 3
  10. 2026Test-Time Safety Alignmentcited by 2
  11. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  12. 2025Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractorscited by 1
11 more
  1. 2026GUARD-SLM: Token Activation-Based Defense Against Jailbreak Attacks for Small Language Modelscited by 1
  2. 2026D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewritingcited by 1
  3. 2025SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Modelscited by 0
  4. 2026Mitigating Many-shot Jailbreak Attacks with One Single Demonstrationcited by 0
  5. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-offcited by 0
  6. 2026EVA: Editing for Versatile Alignment against Jailbreakscited by 0
  7. 2026Babel: Jailbreaking Safety Attention via Obfuscation Distribution Optimized Samplingcited by 0
  8. 2026SafeSpec: Fast and Safe LLM via Dynamic Reflective Samplingcited by 0
  9. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systemscited by 0
  10. 2026Moving the Safety Barrier: Dynamic Routing Adaptive Alignment Against White-Box Attackscited by 0
  11. 2026Safety Cost of Steering Vectors Is Separable and Reduciblecited by 0