mlsanityData to 7 Oct 2026

Model Merging and Safety Alignment: One Bad Model Spoils the Bunch

arXiv 2406.14563 ↗First posted 20 Jun 2024cs.CL cs.AI cs.LGcited by 41

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Merging Large Language Models (LLMs) is a cost-effective technique for combining multiple expert LLMs into a single versatile model, retaining the expertise of the original ones. However, current approaches often overlook the importance of safety alignment during merging, leading to highly misaligned models. This work investigates the effects of model merging on alignment. We evaluate several popular model merging techniques, demonstrating that existing methods do not only transfer domain expertise but also propagate misalignment. We propose a simple two-step approach to address this problem: (i) generating synthetic safety and domain-specific data, and (ii) incorporating these generated data into the optimization process of existing data-aware model merging techniques. This allows us to treat alignment as a skill that can be maximized in the resulting merged LLM. Our experiments illustrate the effectiveness of integrating alignment-related data during merging, resulting in models that excel in both domain expertise and alignment.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 44 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

34 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2020Aligning AI With Shared Human Valuescited by 706
19 more
  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  4. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  5. 2023Llemma: An Open Language Model For Mathematicscited by 385
  6. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  7. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  8. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  10. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  11. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  12. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  13. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  14. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  15. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  16. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  17. 2022Self-Prompting Large Language Models for Zero-Shot Open-Domain QAcited by 39
  18. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  19. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 20; 2026: 15.

41 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  4. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  5. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  6. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  7. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  8. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  9. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  10. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  11. 2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
  12. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
29 more
  1. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 12
  2. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  3. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  4. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  5. 2025HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Mergingcited by 6
  6. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 5
  7. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  8. 2025Efficient Compositional Multi-tasking for On-device Large Language Modelscited by 4
  9. 2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
  10. 2025Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacycited by 3
  11. 2025K-Merge: Online Continual Merging of Adapters for On-device Large Language Modelscited by 3
  12. 2026The Defense Trilemma: Why Prompt Injection Defense Wrappers Fail?cited by 2
  13. 2025Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defensescited by 1
  14. 2025Exploring and Leveraging Class Vectors for Classifier Editingcited by 1
  15. 2025AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraintscited by 1
  16. 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
  17. 2026Resolving Interference (RI): Disentangling Models for Improved Model Mergingcited by 1
  18. 2026Analytic Drift Resister for Non-Exemplar Continual Graph Learningcited by 1
  19. 2026Unlocking the Potential of Continual Model Merging: An ODE Perspectivecited by 1
  20. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  21. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  22. 2026Data-driven Clustering and Merging of Adapters for On-device Large Language Modelscited by 0
  23. 2026Stemma: Induced Decision Regions Reveal LLM Provenancecited by 0
  24. 2026Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognitioncited by 0
  25. 2026When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMscited by 0
  26. 2026DiffSafeMerge: Mitigating Backdoor Inheritance in Diffusion Model Mergingcited by 0
  27. 2026A Single Suffix to Break Them All: Basin-Aware Jailbreaks for Merged Model Familiescited by 0
  28. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  29. 2026Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs
  2. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  3. 2026Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
  4. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  5. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture
  6. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  7. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges