mlsanityData to 14 Oct 2026

Merging Models with Fisher-Weighted Averaging

arXiv 2111.09832 ↗First posted 18 Nov 2021cs.LGcited by 621

Not on the map. Why?

Cited by 27 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities. In this paper, we take the perspective that this "merging" operation can be seen as choosing parameters that approximately maximize the joint likelihood of the posteriors of the models' parameters. Computing a simple average of the models' parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors. We develop an alternative merging procedure based on the Laplace approximation where we approximate each model's posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. We first show that our "Fisher merging" technique provides a performance boost in settings where simple parameter averaging is currently used -- specifically, robust fine-tuning and model ensembling. Then, we compare merging to standard gradient-based transfer learning and demonstrate that merging enables a fundamentally different method for transferring capabilities across models. Specifically, we show that Fisher merging is competitive with gradient-based transfer learning approaches (while being significantly cheaper) in intermediate-task training and domain-adaptive pre-training. We also show that our merging procedure makes it possible to combine models in previously unexplored ways. We release our code to facilitate future research into methods for merging models.

Key facts. Cited by 621 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

42 of 73 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2018Deep contextualized word representationscited by 2,290
30 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2014How transferable are features in deep neural networks?cited by 1,225
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
  13. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  14. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  15. 2020Federated Learning with Matched Averagingcited by 564
  16. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  17. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  18. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  19. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  20. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  21. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  22. 2015Semi-supervised Sequence Learningcited by 198
  23. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  24. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  25. 2020Exploring and Predicting Transferability across NLP Taskscited by 128
  26. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  27. 2018An elementary introduction to information geometrycited by 76
  28. 2019Where is the Information in a Deep Neural Network?cited by 42
  29. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  30. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 20; 2023: 67; 2024: 159; 2025: 204; 2026: 170.

621 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2022Editing Models with Task Arithmeticcited by 1,120
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  5. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  6. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  7. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  8. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  9. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  11. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
38 more
  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  2. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  3. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  4. 2022Patching open-vocabulary models by interpolating weightscited by 190
  5. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  6. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  7. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  8. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  9. 2024Towards Safer Large Language Models through Machine Unlearningcited by 139
  10. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  11. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  12. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  13. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  14. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  15. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  16. 2024Model merging with SVD to tie the Knotscited by 122
  17. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  18. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  19. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  20. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  21. 2022Fusing finetuned models for better pretrainingcited by 110
  22. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  23. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  24. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  25. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  26. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  27. 2023Deep Model Fusion: A Surveycited by 91
  28. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  29. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  30. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  31. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  32. 2023Modular Deep Learningcited by 83
  33. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  34. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  35. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  36. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  37. 2024Parameter Competition Balancing for Model Mergingcited by 72
  38. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67

50 most cited of 621 All 621 (JSON)