Data to 7 Oct 2026

A safety realignment framework via subspace-oriented model fusion for large language models

arXiv 2405.09055 ↗First posted 15 May 2024cs.CLcited by 40

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

The current safeguard mechanisms for large language models (LLMs) are indeed susceptible to jailbreak attacks, making them inherently fragile. Even the process of fine-tuning on apparently benign data for downstream tasks can jeopardize safety. One potential solution is to conduct safety fine-tuning subsequent to downstream fine-tuning. However, there's a risk of catastrophic forgetting during safety fine-tuning, where LLMs may regain safety measures but lose the task-specific knowledge acquired during downstream fine-tuning. In this paper, we introduce a safety realignment framework through subspace-oriented model fusion (SOMF), aiming to combine the safeguard capabilities of initially aligned model and the current fine-tuned model into a realigned model. Our approach begins by disentangling all task vectors from the weights of each fine-tuned model. We then identify safety-related regions within these vectors by subspace masking techniques. Finally, we explore the fusion of the initial safely aligned LLM with all task vectors based on the identified safety subspace. We validate that our safety realignment framework satisfies the safety requirements of a single fine-tuned model as well as multiple models during their fusion. Our findings confirm that SOMF preserves safety without notably compromising performance on downstream tasks, including instruction following in Chinese, English, and Hindi, as well as problem-solving capabilities in Code and Math.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 49 of its 57 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

49 of 57 references.

Closest ancestors · 2

Same direction, newest first

Other references · 47

2022Training language models to follow instructions with human feedbackcited by 15,712
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2021Evaluating Large Language Models Trained on Codecited by 8,053
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2016Overcoming catastrophic forgetting in neural networkscited by 4,718
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366

35 more

2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
2022Editing Models with Task Arithmeticcited by 1,076
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
2017A Survey on Multi-Task Learningcited by 838
2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
2024Self-Rewarding Language Modelscited by 654
2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
2021Merging Models with Fisher-Weighted Averagingcited by 613
2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
2023Pretraining Language Models with Human Preferencescited by 245
2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
2024ReFT: Representation Finetuning for Language Modelscited by 186
2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
2014Neural Variational Inference and Learning in Belief Networkscited by 116
2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
2024Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Modelcited by 25
2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
2024Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillationcited by 18
2024Advancing the Robustness of Large Language Models through Self-Denoised Smoothingcited by 15
2022When does Parameter-Efficient Transfer Learning Work for Machine Translation?cited by 7
2023HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 18; 2026: 6.

40 papers.

Sort:

By area:

2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25

28 more

2024Locking Down the Finetuned LLMs Safetycited by 22
2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
2025Self-Destructive Language Modelcited by 10
2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2
2026A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Lawscited by 0
2026Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognitioncited by 0
2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0