# SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging

arXiv: https://arxiv.org/abs/2503.17239 · HTML: https://mlsanity.com/paper/2503.17239/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 46)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [Removing RLHF Protections in GPT-4 via Fine-Tuning](https://mlsanity.com/paper/2311.05553/) (2023)
- [Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models](https://mlsanity.com/paper/2402.02207/) (2024)
- [Tamper-Resistant Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2408.00761/) (2024)
- [Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models](https://mlsanity.com/paper/2405.16833/) (2024)
- [What is in Your Safe Data? Identifying Benign Data that Breaks Safety](https://mlsanity.com/paper/2404.01099/) (2024)

## Built on by (most cited first, 25 of 25)

- [Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment](https://mlsanity.com/paper/2507.18631/) (2025)
- [Gradient Surgery for Safe LLM Fine-Tuning](https://mlsanity.com/paper/2508.07172/) (2025)
- [Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recovered](https://mlsanity.com/paper/2510.08464/) (2025)
- [Speculative Safety-Aware Decoding](https://mlsanity.com/paper/2508.17739/) (2025)
- [Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study](https://mlsanity.com/paper/2505.14185/) (2025)
- [When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment](https://mlsanity.com/paper/2506.07452/) (2025)
- [Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink](https://mlsanity.com/paper/2602.05228/) (2026)
- [Panini: Continual Learning in Token Space via Structured Memory](https://mlsanity.com/paper/2602.15156/) (2026)
- [Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints](https://mlsanity.com/paper/2604.12384/) (2026)
- [Robust Policy Optimization to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/2602.08813/) (2026)
- [Multi-Level Safety Continual Projection for Fine-Tuned Large Language Models without Retraining](https://mlsanity.com/paper/2508.09190/) (2025)
- [Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI](https://mlsanity.com/paper/2512.01039/) (2025)
- [Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment](https://mlsanity.com/paper/2601.07200/) (2026)
- [GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning](https://mlsanity.com/paper/2603.10243/) (2026)
- [AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints](https://mlsanity.com/paper/2512.16245/) (2025)
- [SafeGene: Reusable Adapters for Transferable Safety Alignment](https://mlsanity.com/paper/2606.06519/) (2026)
- [From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models](https://mlsanity.com/paper/2609.19553/) (2026)
- [SafeCOMM: A Study on Safety Degradation in Fine-Tuned Telecom Large Language Models](https://mlsanity.com/paper/2506.00062/) (2025)
- [Before We Trust Them: Decision-Making Failures in Navigation of Foundation Models](https://mlsanity.com/paper/2601.05529/) (2026)
- [CSULoRA: Closest Safe Update Low-Rank Adaptation](https://mlsanity.com/paper/2605.30640/) (2026)
- [When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs](https://mlsanity.com/paper/2608.08542/) (2026)
- [DiffSafeMerge: Mitigating Backdoor Inheritance in Diffusion Model Merging](https://mlsanity.com/paper/2608.09445/) (2026)
- [SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs](https://mlsanity.com/paper/2609.22153/) (2026)
- [Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation](https://mlsanity.com/paper/2609.31186/) (2026)
- [Aligned Data Can Induce Misalignment via Context Confusion](https://mlsanity.com/paper/2609.38379/) (2026)

## Abstract (arXiv)

Fine-tuning large language models (LLMs) is a common practice to adapt generalist models to specialized domains. However, recent studies show that fine-tuning can erode safety alignment, causing LLMs to respond to harmful or unethical prompts. Many methods to realign safety have been proposed, but often introduce custom algorithms that are difficult to implement or compromise task utility. In this work, we propose SafeMERGE, a lightweight, post-fine-tuning framework that restores safety while maintaining downstream performance. SafeMERGE selectively merges fine-tuned with safety-aligned model layers only when they deviate from safe behavior, measured by a cosine similarity criterion. Across four LLMs and several tasks, SafeMERGE consistently reduces harmful outputs compared to other defenses, with negligible or even positive impact on utility. Our results demonstrate that selective, layer-wise merging offers a robust safeguard against the inadvertent loss of safety during fine-tuning, establishing SafeMERGE as a simple yet effective post-fine-tuning defense.