mlsanityData to 14 Oct 2026

Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!

arXiv 2402.12343 ↗First posted 19 Feb 2024cs.CL cs.AI cs.LGcited by 37

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) undergo safety alignment to ensure safe conversations with humans. However, this paper introduces a training-free attack method capable of reversing safety alignment, converting the outcomes of stronger alignment into greater potential for harm by accessing only LLM output token distributions. Specifically, our method achieves this reversal by contrasting the output token distribution of a safety-aligned language model (e.g., Llama-2-chat) against its pre-trained version (e.g., Llama-2), so that the token predictions are shifted towards the opposite direction of safety alignment. We name this method emulated disalignment (ED) because sampling from this contrastive distribution provably emulates the result of fine-tuning to minimize a safety reward. Our experiments with ED across three evaluation datasets and four model families (Llama-1, Llama-2, Mistral, and Alpaca) show that ED doubles the harmfulness of pre-trained models and outperforms strong baselines, achieving the highest harmful rates in 43 out of 48 evaluation subsets by a large margin. Eventually, given ED's reliance on language model output token distributions, which particularly compromises open-source models, our findings highlight the need to reassess the open accessibility of language models, even if they have been safety-aligned. Code is available at https://github.com/ZHZisZZ/emulated-disalignment.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 39 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

26 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
11 more
  1. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  2. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  3. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  4. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  6. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  7. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  9. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
  10. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  11. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 20; 2026: 7.

37 papers.

Sort:

By area:

  1. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  4. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 56
  5. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 41
  6. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 35
  7. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  8. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 30
  9. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networkscited by 29
  10. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  11. 2024Inference-Time Language Model Alignment via Integrated Value Guidancecited by 19
  12. 2024Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Modelscited by 13
25 more
  1. 2025Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillationcited by 11
  2. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 10
  3. 2024Jailbreaking? One Step Is Enough!cited by 9
  4. 2025Beyond Surface-Level Patterns: An Essence-Driven Defense Framework Against Jailbreak Attacks in LLMscited by 9
  5. 2025LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactionscited by 8
  6. 2025The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agentscited by 7
  7. 2025NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Modelscited by 6
  8. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 6
  9. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  10. 2025The Right to AIcited by 3
  11. 2025JULI: Jailbreak Large Language Models by Self-Introspectioncited by 3
  12. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  13. 2025Contrastive Weak-to-strong Generalizationcited by 2
  14. 2025Self-Instruct Few-Shot Jailbreaking: Decompose the Attack into Pattern and Behavior Learningcited by 1
  15. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspectivecited by 1
  16. 2025PD3F: A Pluggable and Dynamic DoS-Defense Framework Against Resource Consumption Attacks Targeting Large Language Modelscited by 1
  17. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  18. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0
  19. 2025SafeConstellations: Mitigating Over-Refusals in LLMs Through Task-Aware Representation Steeringcited by 0
  20. 2025Matching Ranks Over Probability Yields Truly Deep Safety Alignmentcited by 0
  21. 2026Silencing the Guardrails: Inference-Time Jailbreaking via Dynamic Contextual Representation Ablationcited by 0
  22. 2026Refusal-Gated Decoding: Preserving Refusal Behavior Under High-Temperature Samplingcited by 0
  23. 2026BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditingcited by 0
  24. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  25. 2026Controlled Decoding Attacks on Black-Box LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  3. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  4. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  5. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
  6. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
  7. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  8. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals