mlsanityData to 7 Oct 2026

Safety Alignment Should Be Made More Than Just a Few Tokens Deep

arXiv 2406.05946 ↗First posted 10 Jun 2024cs.CR cs.AIcited by 450

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The safety alignment of current Large Language Models (LLMs) is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We argue that many of these vulnerabilities are related to a shared underlying issue: safety alignment can take shortcuts, wherein the alignment adapts a model's generative distribution primarily over only its very first few output tokens. We refer to this issue as shallow safety alignment. In this paper, we present case studies to explain why shallow safety alignment can exist and provide evidence that current aligned LLMs are subject to this issue. We also show how these findings help explain multiple recently discovered vulnerabilities in LLMs, including the susceptibility to adversarial suffix attacks, prefilling attacks, decoding parameter attacks, and fine-tuning attacks. Importantly, we discuss how this consolidated notion of shallow safety alignment sheds light on promising research directions for mitigating these vulnerabilities. For instance, we show that deepening the safety alignment beyond just the first few tokens can often meaningfully improve robustness against some common exploits. Finally, we design a regularized finetuning objective that makes the safety alignment more persistent against fine-tuning attacks by constraining updates on initial tokens. Overall, we advocate that future safety alignment should be made more than just a few tokens deep.

Key facts. Cited by 450 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 88 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

57 of 88 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  2. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  4. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  5. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  7. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  10. 2023Are aligned neural networks adversarially aligned?26 Jun
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
34 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  9. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  11. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  13. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  14. 2019Control Barrier Functions: Theory and Applicationscited by 332
  15. 2023AI Alignment: A Comprehensive Surveycited by 256
  16. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  17. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  18. 2023Mitigating the Alignment Tax of RLHFcited by 181
  19. 2021Alignment of Language Agentscited by 167
  20. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  21. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  22. 2023Controlled Decoding from Language Modelscited by 133
  23. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  24. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  25. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  26. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  27. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  28. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  29. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  30. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attackscited by 48
  31. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  32. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 35
  33. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 35
  34. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 207; 2026: 211.

450 papers.

Sort:

By area:

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  2. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  3. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  4. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 118
  7. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  8. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  9. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  10. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  11. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  12. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
38 more
  1. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  3. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  4. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  5. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  6. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  7. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  8. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 52
  9. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  10. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 50
  11. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  12. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  13. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  14. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  15. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  16. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 41
  17. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  18. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  19. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  20. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  21. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  22. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  23. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  24. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  25. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  26. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  27. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  28. 2025Representation Bending for Large Language Model Safetycited by 28
  29. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27
  30. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27
  31. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  32. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  33. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  34. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  35. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 25
  36. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  37. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  38. 2025Steering MoE LLMs via Expert (De)Activationcited by 23

50 most cited of 450 All 450 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  2. 2026Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
  3. 2026Understanding the Effects of Safety Unalignment on Large Language Models
  4. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  5. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  6. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  7. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks
  8. 2023Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack