mlsanityData to 14 Oct 2026

Will it Merge? On The Causes of Model Mergeability

arXiv 2601.06672 ↗First posted 10 Jan 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining. However, the factors that determine whether merging will succeed or fail remain poorly understood. In this work, we investigate why specific models are merged better than others. To do so, we propose a concrete, measurable definition of mergeability. We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models fine-tuned on instances that the base model knows better are more mergeable than models fine-tuned on instances that the base model struggles with. Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 10 Jan 2026.

Builds on

21 of 25 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  6. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  7. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  8. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  9. 2024Model merging with SVD to tie the Knotscited by 122
  10. 2022Fusing finetuned models for better pretrainingcited by 110
  11. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  12. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
9 more
  1. 2017Identifying beneficial task relations for multi-task learning in deep neural networkscited by 59
  2. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  3. 2024What Matters for Model Merging at Scale?cited by 55
  4. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  5. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 33
  6. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  7. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  8. 2023TaskWeb: Selecting Better Source Tasks for Multi-task NLPcited by 11
  9. 2024Collective Model Intelligence Requires Compatible Specializationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 7
  2. 2026NEX: Neuron Explore-Exploit Scoring for Label-Free Chain-of-Thought Selection and Model Rankingcited by 4
  3. 2026Model Merging: Foundations and Algorithmscited by 0
  4. 2026Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMscited by 0