mlsanityData to 7 Oct 2026

Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink

arXiv 2602.05228 ↗First posted 5 Feb 2026cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Harmful fine-tuning can invalidate safety alignment of large language models, exposing significant safety risks. In this paper, we utilize the attention sink mechanism to mitigate harmful fine-tuning. Specifically, we first measure a statistic named sink divergence for each attention head and observe that different attention heads exhibit two different signs of sink divergence. To understand its safety implications, we conduct experiments and find that the number of attention heads of positive sink divergence increases along with the increase of the model's harmfulness when undergoing harmful fine-tuning. Based on this finding, we propose a separable sink divergence hypothesis -- attention heads associating with learning harmful patterns during fine-tuning are separable by their sign of sink divergence. Based on the hypothesis, we propose a fine-tuning-stage defense, dubbed Surgery. Surgery utilizes a regularizer for sink divergence suppression, which steers attention heads toward the negative sink divergence group, thereby reducing the model's tendency to learn and amplify harmful patterns. Extensive experiments demonstrate that Surgery improves defense performance by 5.90%, 11.25%, and 9.55% on the BeaverTails, HarmBench, and SorryBench benchmarks, respectively. Source code is available on https://github.com/Lslland/Surgery.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 132 of its 167 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

132 of 167 references.

Closest ancestors · 37

Same direction, newest first

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance5 Jan
  2. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning10 Dec
  3. 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuning22 Nov
  4. 2025HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment17 Oct
  5. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space16 Oct
  6. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning11 Oct
  7. 2025Gradient Surgery for Safe LLM Fine-Tuning10 Aug
  8. 2025SDD: Self-Degraded Defense against Malicious Fine-tuning27 Jul
  9. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment24 Jul
  10. 2025Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs21 Jun
  11. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models19 Jun
  12. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning4 Jun
25 more
  1. 2025Shape it Up! Restoring LLM Safety during Finetuning22 May
  2. 2025Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?14 Apr
  3. 2025LookAhead Tuning: Safer Language Models via Partial Answer Previews24 Mar
  4. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging21 Mar
  5. 2025The dark deep side of DeepSeek: Fine-tuning attacks against the safety alignment of CoT-enabled models3 Feb
  6. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  7. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning17 Dec
  8. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models15 Dec
  9. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  10. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models11 Oct
  11. 2024Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models1 Oct
  12. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  13. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  14. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  15. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  16. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models12 Jun
  17. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  18. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  19. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  20. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  21. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  22. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  23. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  24. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  25. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 95

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  10. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  11. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  12. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
38 more
  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  2. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  3. 2025Why do LLMs attend to the first token?cited by 122
  4. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  5. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  6. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  7. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  8. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  9. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  10. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  11. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  12. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  13. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  14. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  15. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  16. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 52
  17. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  18. 2024Open Problems in Technical AI Governancecited by 45
  19. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  20. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 37
  21. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 35
  22. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  23. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  24. 2023Exploiting Novel GPT-4 APIscited by 27
  25. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  26. 2024Scaling Trends for Data Poisoning in LLMscited by 26
  27. 2025What are you sinking? A geometric approach on attention sinkcited by 26
  28. 2024RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Responsecited by 24
  29. 2024Locking Down the Finetuned LLMs Safetycited by 22
  30. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  31. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  32. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biasescited by 21
  33. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  34. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 20
  35. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 20
  36. 2023Making Harmful Behaviors Unlearnable for Large Language Modelscited by 19
  37. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  38. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMscited by 18

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  2. 2026Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformerscited by 2
  3. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  2. 2025In-Training Defenses against Emergent Misalignment in Language Models
  3. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
  4. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  5. 2025Safety Alignment Should Be Made More Than Just A Few Attention Heads
  6. 2026Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis
  7. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
  8. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence