mlsanityData to 14 Oct 2026

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces

arXiv 2502.04959 ↗First posted 7 Feb 2025cs.LGcited by 96

Not on the map. Why?

Abstract (arXiv)

Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices -- weight update matrices applied to a pre-trained model -- that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training. Code is available at https://github.com/danielm1405/iso-merging .

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

40 of 59 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2016Layer Normalizationcited by 5,669
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
28 more
  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  3. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  7. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  8. 2018Deep Learning for Classical Japanese Literaturecited by 422
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  10. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  11. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  12. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  13. 2022Patching open-vocabulary models by interpolating weightscited by 190
  14. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
  15. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  16. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  17. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  18. 2024Model merging with SVD to tie the Knotscited by 122
  19. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  20. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  21. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  22. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  23. 2023Deep Model Fusion: A Surveycited by 91
  24. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  25. 2024Parameter Competition Balancing for Model Mergingcited by 72
  26. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  27. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  28. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 71.

96 papers.

Sort:

By area:

  1. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 29
  2. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 18
  3. 2025RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Mergingcited by 11
  4. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 10
  5. 2025AdaRank: Adaptive Rank Pruning for Enhanced Model Mergingcited by 9
  6. 2025Subspace-Boosted Model Mergingcited by 9
  7. 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunkingcited by 8
  8. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  9. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 7
  10. 2026Model Merging in the Essential Subspacecited by 7
  11. 2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
  12. 2025Model Unmerging: Making Your Models Unmergeable for Secure Model Sharingcited by 6
38 more
  1. 2026Transporting Task Vectors across Different Architectures without Trainingcited by 6
  2. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 6
  3. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 5
  4. 2025Rethinking Layer-wise Model Merging through Chain of Mergescited by 5
  5. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  6. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  7. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  8. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 4
  9. 2025Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Modelscited by 4
  10. 2026Orthogonal Model Mergingcited by 4
  11. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  12. 2026Decoupled DiLoCo for Resilient Distributed Pre-trainingcited by 4
  13. 2025Towards Efficient Multimodal Unified Reasoning Model via Model Mergingcited by 3
  14. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
  15. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3
  16. 2025Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defensescited by 2
  17. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  18. 2025Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Mergingcited by 2
  19. 2025MAGIC: Achieving Superior Model Merging via Magnitude Calibrationcited by 2
  20. 2025Model Merging via Multi-Teacher Knowledge Distillationcited by 2
  21. 2026Bridging Domains through Subspace-Aware Model Mergingcited by 2
  22. 2026Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosiscited by 2
  23. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  24. 2026Zero-Shot Quantization via Weight-Space Arithmeticcited by 2
  25. 2026PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Mergingcited by 2
  26. 2026Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Mergingcited by 2
  27. 2026PACT: Preserving Anchored Cores in Task-vectors for Model Mergingcited by 2
  28. 2025DisTaC: Conditioning Task Vectors via Distillation for Robust Model Mergingcited by 1
  29. 2025Weight Weaving: Parameter Pooling for Data-Free Model Mergingcited by 1
  30. 2025From Coefficients to Directions: Rethinking Model Merging with Directional Alignmentcited by 1
  31. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 1
  32. 2026Learning Not To Interfere: Adapting Experts For Model Mergingcited by 1
  33. 2026Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimizationcited by 1
  34. 2026Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problemcited by 1
  35. 2026Generalizing the Geometry of Model Merging Through Frechet Averagescited by 1
  36. 2026Dynamic Model Merging Made Slimcited by 1
  37. 2026Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmeticcited by 1
  38. 2026Model Merging on Loss Landscape: A Geometry Perspectivecited by 1

50 most cited of 96 All 96 (JSON)