mlsanityData to 7 Oct 2026

Safety Layers in Aligned Large Language Models: The Key to LLM Security

arXiv 2408.17003 ↗First posted 30 Aug 2024cs.CR cs.AIcited by 126

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Aligned LLMs are secure, capable of recognizing and refusing to answer malicious questions. However, the role of internal parameters in maintaining such security is not well understood yet, further these models can be vulnerable to security degradation when subjected to fine-tuning attacks. To address these challenges, our work uncovers the mechanism behind security in aligned LLMs at the parameter level, identifying a small set of contiguous layers in the middle of the model that are crucial for distinguishing malicious queries from normal ones, referred to as ``safety layers". We first confirm the existence of these safety layers by analyzing variations in input vectors within the model's internal layers. Additionally, we leverage the over-rejection phenomenon and parameters scaling analysis to precisely locate the safety layers. Building on these findings, we propose a novel fine-tuning approach, Safely Partial-Parameter Fine-Tuning (SPPFT), that fixes the gradient of the safety layers during fine-tuning to address the security degradation. Our experiments demonstrate that the proposed approach can significantly preserve LLM security while maintaining performance and reducing computational resources compared to full fine-tuning.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 46 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

34 of 46 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  4. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  5. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  6. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
  7. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  8. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2020Aligning AI With Shared Human Valuescited by 706
10 more
  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  2. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  3. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  4. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  5. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  6. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  7. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  8. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  9. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  10. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 66; 2026: 51.

126 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  4. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  6. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  7. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  8. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  9. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  10. 2025Representation Bending for Large Language Model Safetycited by 28
  11. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  12. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
38 more
  1. 2025Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuningcited by 22
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  3. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  4. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  5. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  6. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  7. 2024Superficial Safety Alignment Hypothesiscited by 14
  8. 2025Depth Gives a False Sense of Privacy: LLM Internal States Inversioncited by 14
  9. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  10. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  11. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  12. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  13. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  14. 2025Silenced Biases: The Dark Side LLMs Learned to Refusecited by 10
  15. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representationscited by 10
  16. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMscited by 9
  17. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 9
  18. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  19. 2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8
  20. 2024How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?cited by 8
  21. 2025Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial Statescited by 8
  22. 2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
  23. 2025LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMscited by 7
  24. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 7
  25. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  26. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  27. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  28. 2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 6
  29. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  30. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  31. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  32. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 5
  33. 2025An Investigation of Memorization Risk in Healthcare Foundation Modelscited by 5
  34. 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuningcited by 5
  35. 2024Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weightingcited by 4
  36. 2025Efficient LLM Moderation with Multi-Layer Latent Prototypescited by 4
  37. 2025Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Modelscited by 4
  38. 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 4

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  2. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  3. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
  4. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  5. 2025StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Models
  6. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  7. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  8. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals