mlsanityData to 14 Oct 2026

Subspace-Boosted Model Merging

arXiv 2506.16506 ↗First posted 19 Jun 2025cs.LG cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Model merging enables the combination of multiple specialized expert models into a single model capable of performing multiple tasks. However, the benefits of merging an increasing amount of specialized experts generally lead to diminishing returns and reduced overall performance gains. In this work, we empirically and theoretically analyze this limitation, proving that for Task Arithmetic-based methods, as more experts are merged, the common information dominates the task-specific information, leading to inevitable rank collapse. To mitigate this issue, we introduce Subspace Boosting, which operates on the singular value decomposed task vector space and maintains task vector ranks. Subspace Boosting raises merging efficacy for up to 20 experts by large margins of more than 10% when evaluated on both vision and language benchmarks. Moreover, we propose employing Higher-Order Generalized Singular Value Decomposition to quantify task similarity, offering a new interpretable perspective on model merging. Code and models are available at https://github.com/ronskoro/Subspace-Boosting.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 69 of its 86 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

69 of 86 references.

References · 69

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  12. 2022Editing Models with Task Arithmeticcited by 1,120
38 more
  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  2. 2021Robust fine-tuning of zero-shot modelscited by 797
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  5. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  9. 2020What is being transferred in transfer learning?cited by 368
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  11. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  12. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  13. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  14. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  15. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  16. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  17. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  18. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  19. 2024Model merging with SVD to tie the Knotscited by 122
  20. 2020Revisiting Training Strategies and Generalization Performance in Deep Metric Learningcited by 100
  21. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  22. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  23. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  24. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  25. 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
  26. 2013Deep Learning of Representations: Looking Forwardcited by 84
  27. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  28. 2025Transformers without Normalizationcited by 70
  29. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  30. 2024What Matters for Model Merging at Scale?cited by 55
  31. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  32. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  33. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  34. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  35. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  36. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  37. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
  38. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Steering Language Models with Weight Arithmeticcited by 20
  2. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  3. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  4. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 1
  5. 2026Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancingcited by 1
  6. 2026Training-free Task Classification for Multi-Task Model Mergingcited by 1
  7. 2026Functionality-Oriented LLM Merging on the Fisher--Rao Manifoldcited by 0
  8. 2026Interpretability-Guided Layer Selection over Subspace Projection: SAEs as Stethoscopes, Not Scalpels, for Raw Task Vector Model Editingcited by 0
  9. 2026Federated Subspace Guided Vision-Language-Action Policy Distillation for Non-IID Multi-Robot Manipulationcited by 0