mlsanityData to 14 Oct 2026

Model Merging by Uncertainty-Based Gradient Matching

arXiv 2310.12808 ↗First posted 19 Oct 2023cs.LG cs.AI cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters. Code available here.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

39 of 68 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
27 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  2. 2022Editing Models with Task Arithmeticcited by 1,120
  3. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  9. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  11. 2014New insights and perspectives on the natural gradient methodcited by 443
  12. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  13. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  14. 2020What is being transferred in transfer learning?cited by 368
  15. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  16. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  17. 2019Practical Deep Learning with Bayesian Principlescited by 158
  18. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  19. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  20. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  21. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 109
  22. 2015Distributed Gaussian Processescited by 84
  23. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 55
  24. 2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
  25. 2021Knowledge-Adaptation Priorscited by 20
  26. 2022Revisiting Checkpoint Averaging for Neural Machine Translationcited by 13
  27. 2022Bayesian data fusion with shared priorscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 27; 2025: 35; 2026: 13.

77 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  3. 2024Model merging with SVD to tie the Knotscited by 122
  4. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  5. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  6. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  7. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  8. 2024What Matters for Model Merging at Scale?cited by 55
  9. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  10. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  11. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  12. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
38 more
  1. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28
  2. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  3. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  4. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  5. 2024Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reductioncited by 18
  6. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  7. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  8. 2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 14
  9. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRAcited by 14
  10. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 14
  11. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  12. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 13
  13. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  14. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  15. 2025Dynamic Fisher-weighted Model Merging via Bayesian Optimizationcited by 12
  16. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  17. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 11
  18. 2025STAR: Spectral Truncation and Rescale for Model Mergingcited by 11
  19. 2025RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Mergingcited by 11
  20. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  21. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  22. 2024Rethinking Weight-Averaged Model-mergingcited by 8
  23. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  24. 2024Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Modelscited by 7
  25. 2025Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learningcited by 7
  26. 2025On Task Vectors and Gradientscited by 7
  27. 2025Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDPcited by 7
  28. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 7
  29. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  30. 2025Knowledge Adaptation as Posterior Correctioncited by 6
  31. 2024Variational Low-Rank Adaptation Using IVONcited by 5
  32. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 5
  33. 2025Rethinking Layer-wise Model Merging through Chain of Mergescited by 5
  34. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  35. 2026FeatCal: Feature Calibration for Post-Merging Modelscited by 5
  36. 2025Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMscited by 4
  37. 2025Model Merging with Functional Dual Anchorscited by 4
  38. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4

50 most cited of 77 All 77 (JSON)