mlsanityData to 14 Oct 2026

Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning

arXiv 2512.10150 ↗First posted 10 Dec 2025cs.CL cs.AIcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The safety alignment of large language models (LLMs) is becoming increasingly important with their democratization. In this paper, we study the safety degradation that comes with adapting LLMs to new tasks. We attribute this safety compromise to catastrophic forgetting and frame the problem of preserving safety when fine-tuning as a continual learning (CL) problem. We consider the fine-tuning-as-a-service setup where the user uploads their data to a service provider to get a customized model that excels on the user's selected task. We adapt several CL approaches from the literature and systematically evaluate their ability to mitigate safety degradation. These include regularization-based, memory-based, and model merging approaches. We consider two scenarios, (1) benign user data and (2) poisoned user data. Our results demonstrate that CL approaches consistently achieve lower attack success rates than standard fine-tuning. Among these, DER outperforms both other CL methods and existing safety-preserving baselines while maintaining task utility. These findings generalize across three downstream tasks (GSM8K, SST2, Code) and three model families (LLaMA2-7B, Mistral-7B, Gemma-2B), establishing CL as a practical solution to preserve safety.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 50 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

45 of 50 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025LookAhead Tuning: Safer Language Models via Partial Answer Previews24 Mar
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  4. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  5. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation24 May
  6. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  7. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2016Learning without Forgettingcited by 2,496
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2018Experience Replay for Continual Learningcited by 1,095
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
22 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  4. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  5. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  6. 2021Does Knowledge Distillation Really Work?cited by 190
  7. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  8. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  9. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  10. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  11. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  12. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  13. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  14. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  15. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  16. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  17. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  18. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  19. 2024Can Editing LLMs Inject Harm?cited by 22
  20. 2024Locking Down the Finetuned LLMs Safetycited by 22
  21. 2024A Unified and General Framework for Continual Learningcited by 20
  22. 2021In Defense of the Learning Without Forgetting for Task Incremental Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  2. 2026FSFM: A Biologically-Inspired Framework for Selective Forgetting of Agent Memorycited by 3
  3. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 3
  4. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  5. 2026Beyond Creed: A Non-Identity Safety Condition A Strong Empirical Alternative to Identity Framing in Low-Data LoRA Fine-Tuningcited by 1
  6. 2026SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Modelscited by 0
  7. 2026LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learningcited by 0
  8. 2026Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Studycited by 0
  9. 2026Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluationcited by 0
  10. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  2. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  3. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
  4. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  5. 2026Continual Safety Alignment via Gradient-Based Sample Selection
  6. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  7. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  8. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency