mlsanityData to 14 Oct 2026

SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models

arXiv 2604.17691 ↗First posted 20 Apr 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment in large language models is remarkably shallow: it is concentrated in the first few output tokens and reversible by fine-tuning on as few as 100 adversarial examples. This fragility becomes critical in real-world deployment, where models undergo sequential adaptation across domains such as medicine, law, and code, causing safety guardrails to erode cumulatively. Yet all existing safety-preserving methods target only single-task fine-tuning, leaving the multi-domain sequential setting entirely unaddressed. We introduce SafeAnchor, a framework that anchors safety in place throughout continual adaptation. SafeAnchor first identifies low-rank safety subspaces in LoRA parameter space via Fisher Information eigendecomposition, then constrains domain-specific gradient updates to the orthogonal complement of these subspaces, and finally monitors for residual safety drift with threshold-triggered corrective replay. Evaluated on Llama-2-7B-Chat and Mistral-7B-Instruct across a three-domain pipeline and eight benchmarks, SafeAnchor retains 93.2% of original safety alignment, outperforming all baselines by 18-42 points, while matching unconstrained fine-tuning to within 1.5 points on domain tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 45 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

43 of 45 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection8 Feb
  2. 2025Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation29 Dec
  3. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning10 Dec
  4. 2025Gradient Surgery for Safe LLM Fine-Tuning10 Aug
  5. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable1 Mar
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  8. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  9. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  12. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
2 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2023Mistral 7Bcited by 3,164
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
17 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  5. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  7. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  8. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  9. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  10. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  11. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  12. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  13. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  14. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  15. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 53
  16. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  17. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  2. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  3. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints
  4. 2026Continual Safety Alignment via Gradient-Based Sample Selection
  5. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  6. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  7. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
  8. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks