mlsanityData to 7 Oct 2026

SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance

arXiv 2406.18118 ↗First posted 26 Jun 2024cs.CR cs.CLcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

As the development of large language models (LLMs) rapidly advances, securing these models effectively without compromising their utility has become a pivotal area of research. However, current defense strategies against jailbreak attacks (i.e., efforts to bypass security protocols) often suffer from limited adaptability, restricted general capability, and high cost. To address these challenges, we introduce SafeAligner, a methodology implemented at the decoding stage to fortify defenses against jailbreak attacks. We begin by developing two specialized models: the Sentinel Model, which is trained to foster safety, and the Intruder Model, designed to generate riskier responses. SafeAligner leverages the disparity in security levels between the responses from these models to differentiate between harmful and beneficial tokens, effectively guiding the safety alignment by altering the output token distribution of the target model. Extensive experiments show that SafeAligner can increase the likelihood of beneficial tokens, while reducing the occurrence of harmful ones, thereby ensuring secure alignment with minimal loss to generality.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

33 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance20 Jan
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  6. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2018Hierarchical Neural Story Generationcited by 1,182
14 more
  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  3. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  4. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  5. 2023Detecting Language Model Attacks with Perplexitycited by 411
  6. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  7. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  9. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  10. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  11. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  12. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  13. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  14. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 5.

8 papers.

  1. 2024The Sociolinguistic Foundations of Language Modelingcited by 13
  2. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  3. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Levelcited by 11
  4. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  5. 2025RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplarscited by 2
  6. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  7. 2025MirrorShield: Towards Universal Defense Against Jailbreaks via Entropy-Guided Mirror Craftingcited by 1
  8. 2025Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  2. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment
  3. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  4. 2025Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction
  5. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models
  6. 2026SecureBreak -- A dataset towards safe and secure models
  7. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  8. 2026SafeDream: Safety World Model for Proactive Early Jailbreak Detection