mlsanityData to 14 Oct 2026

SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging

arXiv 2503.17239 ↗First posted 21 Mar 2025cs.CL cs.AIcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Fine-tuning large language models (LLMs) is a common practice to adapt generalist models to specialized domains. However, recent studies show that fine-tuning can erode safety alignment, causing LLMs to respond to harmful or unethical prompts. Many methods to realign safety have been proposed, but often introduce custom algorithms that are difficult to implement or compromise task utility. In this work, we propose SafeMERGE, a lightweight, post-fine-tuning framework that restores safety while maintaining downstream performance. SafeMERGE selectively merges fine-tuned with safety-aligned model layers only when they deviate from safe behavior, measured by a cosine similarity criterion. Across four LLMs and several tasks, SafeMERGE consistently reduces harmful outputs compared to other defenses, with negligible or even positive impact on utility. Our results demonstrate that selective, layer-wise merging offers a robust safeguard against the inadvertent loss of safety during fine-tuning, establishing SafeMERGE as a simple yet effective post-fine-tuning defense.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 67 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

46 of 67 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  4. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation24 May
  5. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  6. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  7. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  8. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  10. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
23 more
  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  6. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  8. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  9. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  10. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  11. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  12. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  13. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  14. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  15. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  16. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  17. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  18. 2023TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledgecited by 63
  19. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  20. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  21. 2024TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specificationscited by 27
  22. 2024Tele-LLMs: A Series of Specialized Large Language Models for Telecommunicationscited by 25
  23. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 15.

25 papers.

Sort:

By area:

  1. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  2. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  3. 2025Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recoveredcited by 8
  4. 2025Speculative Safety-Aware Decodingcited by 7
  5. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  6. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  7. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  8. 2026Panini: Continual Learning in Token Space via Structured Memorycited by 4
  9. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraintscited by 4
  10. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  11. 2025Multi-Level Safety Continual Projection for Fine-Tuned Large Language Models without Retrainingcited by 2
  12. 2025Joint Partitioning and Placement of Foundation Models for Real-Time Edge AIcited by 2
13 more
  1. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
  2. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  3. 2025AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraintscited by 1
  4. 2026SafeGene: Reusable Adapters for Transferable Safety Alignmentcited by 1
  5. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  6. 2025SafeCOMM: A Study on Safety Degradation in Fine-Tuned Telecom Large Language Modelscited by 0
  7. 2026Before We Trust Them: Decision-Making Failures in Navigation of Foundation Modelscited by 0
  8. 2026CSULoRA: Closest Safe Update Low-Rank Adaptationcited by 0
  9. 2026When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMscited by 0
  10. 2026DiffSafeMerge: Mitigating Backdoor Inheritance in Diffusion Model Mergingcited by 0
  11. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  12. 2026Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluationcited by 0
  13. 2026Aligned Data Can Induce Misalignment via Context Confusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  2. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  3. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  4. 2024Locking Down the Finetuned LLMs Safety
  5. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  6. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models
  7. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
  8. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning