mlsanityData to 14 Oct 2026

Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks

arXiv 2506.07356 ↗First posted 9 Jun 2025cs.CLcited by 2

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

While Finetuning-as-a-Service (FaaS) enables customization of Large Language Models (LLMs) using user data, this service is vulnerable to safety degradation when user data includes harmful prompts, a threat known as harmful finetuning attacks. To defend against this, prior work first constructs safety-aligned LLM and then finetunes the LLM on user data. However, we observe that the safety-aligned weights provide weak initialization for downstream task learning, leading to suboptimal safety and utility. Motivated by this limitation, we shift the safe FaaS finetuning paradigm from finetuning safety-aligned weights to finetuning base weights under explicit safety-teacher guidance. Specifically, we propose a Refusal-Teacher (Ref-Teacher)-guided finetuning framework. Our approach directly finetunes the base LLM under the guidance of a safety-aligned Ref-Teacher, which filters harmful prompts from user data and distills safety into the base LLM during finetuning. Extensive experiments demonstrate that our paradigm effectively reduces harmful outputs while improving finetuning performance on user-specific tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 50 of its 65 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

50 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  2. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation3 Jan
  3. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks23 Oct
  4. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  5. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  6. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  7. 2024Improving Alignment and Robustness with Circuit Breakers6 Jun
  8. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  9. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuning23 May
  10. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safety1 Apr
  11. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
26 more
  1. 2019When Does Label Smoothing Help?cited by 988
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  4. 2018Born Again Neural Networkscited by 546
  5. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  6. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  7. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  8. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  9. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  10. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  11. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  12. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  13. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  14. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  15. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  16. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
  17. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  18. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  19. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  20. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 74
  21. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  22. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  23. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  24. 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Modelscited by 25
  25. 2025EXAONE Deep: Reasoning Enhanced Language Modelscited by 21
  26. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Modelscited by 0
  2. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  2. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  3. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
  4. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
  5. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  6. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  7. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  8. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models