mlsanityData to 7 Oct 2026

Revisiting Weight Averaging for Model Merging

arXiv 2412.12153 ↗First posted 11 Dec 2024cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Model merging aims to build a multi-task learner by combining the parameters of individually fine-tuned models without additional training. While a straightforward approach is to average model parameters across tasks, this often results in suboptimal performance due to interference among parameters across tasks. In this paper, we present intriguing results that weight averaging implicitly induces task vectors centered around the weight averaging itself and that applying a low-rank approximation to these centered task vectors significantly improves merging performance. Our analysis shows that centering the task vectors effectively reduces task interference and most of task-specific knowledge is concentrated in the top singular vectors. Our method demonstrates robust and scalable performance on vision benchmarks across varying numbers of tasks and model sizes. Furthermore, we observe that our approach is applicable to natural language processing tasks with competitive performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

35 of 49 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2022Editing Models with Task Arithmeticcited by 1,076
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
23 more
  1. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  3. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  4. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  5. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  6. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  7. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  8. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  9. 2020What is being transferred in transfer learning?cited by 365
  10. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  11. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  12. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  13. 2019Model Fusion via Optimal Transportcited by 217
  14. 2022Patching open-vocabulary models by interpolating weightscited by 187
  15. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  16. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  17. 2022Fusing finetuned models for better pretrainingcited by 110
  18. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  19. 2023Deep Model Fusion: A Surveycited by 89
  20. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  21. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  22. 2021Facial Emotion Recognition: State of the Art Performance on FER2013cited by 20
  23. 2017Combining Weakly and Webly Supervised Learning for Classifying Food Imagescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  2. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  4. 2025When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracycited by 13
  5. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Mergingcited by 12
  6. 2025Subspace-Boosted Model Mergingcited by 9
  7. 2025AdaRank: Adaptive Rank Pruning for Enhanced Model Mergingcited by 8
  8. 2026Kairos: A Regret-Aware Native World-Action Model Stack for Physical AIcited by 8
  9. 2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
  10. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 5
  11. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 5
  12. 2025Towards Efficient Multimodal Unified Reasoning Model via Model Mergingcited by 3
12 more
  1. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3
  2. 2025Defending Unauthorized Model Merging via Dual-Stage Weight Protectioncited by 2
  3. 2025CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learningcited by 1
  4. 2026MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Mergingcited by 1
  5. 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
  6. 2026Resolving Interference (RI): Disentangling Models for Improved Model Mergingcited by 1
  7. 2025Look the Other Way: Designing 'Positive' Molecules with Negative Data via Task Arithmeticcited by 0
  8. 2025R2M: Real-Aware Residual Model Merging for Robust and Generalizable Deepfake Detectioncited by 0
  9. 2026Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Taskscited by 0
  10. 2026Evolutionary Negative Module Pruning for Better LoRA Mergingcited by 0
  11. 2026Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?cited by 0
  12. 2026Riemannian--Lorentz Fusion of Vision Transformers and State-Space Modelscited by 0