mlsanityData to 14 Oct 2026

Shape it Up! Restoring LLM Safety during Finetuning

arXiv 2505.17196 ↗First posted 22 May 2025cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks: even a few harmful examples can compromise safety alignment. A common mitigation strategy is to update the model more strongly on examples deemed safe, while downweighting or excluding those flagged as unsafe. However, because safety context can shift within a single example, updating the model equally on both harmful and harmless parts of a response is suboptimal-a coarse treatment we term static safety shaping. In contrast, we propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content. To enable such fine-grained control during finetuning, we introduce a key insight: guardrail models, traditionally used for filtering, can be repurposed to evaluate partial responses, tracking how safety risk evolves throughout the response, segment by segment. This leads to the Safety Trajectory Assessment of Response (STAR), a token-level signal that enables shaping to operate dynamically over the training sequence. Building on this, we present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families-all without compromising capability on intended tasks. We encourage future safety research to build on dynamic shaping principles for stronger mitigation against evolving finetuning risks. Our code is publicly available at https://github.com/poloclub/star-dss.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 72 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

49 of 72 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  2. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  3. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  4. 2024LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models3 Jul
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  6. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  7. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  8. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  9. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  10. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
4 more
  1. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
21 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  3. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  4. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  5. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  6. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  7. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  8. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  9. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  10. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  11. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  12. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  13. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  14. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  15. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  16. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
  17. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  18. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  19. 2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 17
  20. 2025SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Modelscited by 3
  21. 2021ICPE: An Item Cluster-Wise Pareto-Efficient Framework for Recommendation Debiasingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  2. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  3. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  4. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  5. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  6. 2026Continual Safety Alignment via Gradient-Based Sample Selectioncited by 4
  7. 2025Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AIcited by 0
  8. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  9. 2026Defending Against Harmful Supervision Hidden in Benign Samplescited by 0
  10. 2026Safe Evolution with Circuit Anchorscited by 0
  11. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  2. 2024Locking Down the Finetuned LLMs Safety
  3. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  4. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  5. 2026Guardrails in Logit Space: Safety Token Regularization for LLM Alignment
  6. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  7. 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuning
  8. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning