mlsanityData to 14 Oct 2026

Rethinking Safety in LLM Fine-tuning: An Optimization Perspective

arXiv 2508.12531 ↗First posted 17 Aug 2025cs.LG cs.AIcited by 14

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Fine-tuning language models is commonly believed to inevitably harm their safety, i.e., refusing to respond to harmful user requests, even when using harmless datasets, thus requiring additional safety measures. We challenge this belief through systematic testing, showing that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts. By properly selecting key training hyper-parameters, e.g., learning rate, batch size, and gradient steps, we reduce unsafe model responses from 16% to approximately 5%, as measured by keyword matching, while maintaining utility performance. Based on this observation, we propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance by creating a stable optimization path and retains the original pre-trained model's safety properties. Our experiments on the Llama families across multiple datasets (Dolly, Alpaca, ORCA) demonstrate that safety problems during fine-tuning can largely be avoided without specialized interventions, outperforming existing approaches that require additional safety data while offering practical guidelines for maintaining both model performance and safety during adaptation.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 43 of its 49 references that are in the map. First posted on arXiv 17 Aug 2025.

Builds on

43 of 49 references.

References · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2016Learning without Forgettingcited by 2,496
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
31 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  5. 2022Editing Models with Task Arithmeticcited by 1,120
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  9. 2022Continual Test-Time Domain Adaptationcited by 525
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  11. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  12. 2018Feature Denoising for Improving Adversarial Robustnesscited by 314
  13. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  14. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  15. 2020Bag of Tricks for Adversarial Trainingcited by 200
  16. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  17. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  18. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  19. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  20. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  21. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  22. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  23. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  24. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  25. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 52
  26. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 46
  27. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  28. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
  29. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  30. 2024Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Modelscited by 13
  31. 2022Meta-Learning with Self-Improving Momentum Targetcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generationcited by 5
  2. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  3. 2025Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approachcited by 2
  4. 2025The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMscited by 2
  5. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
  6. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  7. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMscited by 1
  8. 2026Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steeringcited by 1
  9. 2026When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queriescited by 0
  10. 2026HARC: Coupling Harmfulness and Refusal Directions for Robust Safety Alignmentcited by 0
  11. 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuningcited by 0
  12. 2026Event Interaction in Low-Rank Bottlenecks for Temporal Relation Extractioncited by 0
2 more
  1. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Modelscited by 0
  2. 2026SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  2. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  3. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
  4. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  5. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning
  6. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
  7. 2025Gradient Surgery for Safe LLM Fine-Tuning
  8. 2024Locking Down the Finetuned LLMs Safety