mlsanityData to 7 Oct 2026

Git Re-Basin: Merging Models modulo Permutation Symmetries

arXiv 2209.04836 ↗First posted 11 Sep 2022cs.LG cs.AIcited by 460

Not on the map. Why?

Abstract (arXiv)

The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.

Key facts. Cited by 460 papers in the map (data through 7 Oct 2026). Builds on 65 of its 103 references that are in the map. First posted on arXiv 11 Sep 2022.

Builds on

65 of 103 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2018Group Normalizationcited by 1,514
  12. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
38 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2014How transferable are features in deep neural networks?cited by 1,216
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  4. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  11. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  12. 2020Federated Learning with Matched Averagingcited by 564
  13. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  14. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  15. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  16. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  17. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  18. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  19. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  20. 2016Deep Learning without Poor Local Minimacited by 233
  21. 2019Model Fusion via Optimal Transportcited by 217
  22. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  23. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  24. 2020Fast Differentiable Sorting and Rankingcited by 196
  25. 2022Patching open-vocabulary models by interpolating weightscited by 187
  26. 2014Qualitatively characterizing neural network optimization problemscited by 186
  27. 2018Learning Latent Permutations with Gumbel-Sinkhorn Networkscited by 172
  28. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  29. 2021Assessing Generalization of SGD via Disagreementcited by 116
  30. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  31. 2019Stochastic Optimization of Sorting Networks via Continuous Relaxationscited by 111
  32. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  33. 2019Are All Layers Created Equal?cited by 94
  34. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  35. 2021Learning Neural Network Subspacescited by 82
  36. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  37. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  38. 2020SoftSort: A Continuous Relaxation for the argsort Operatorcited by 66

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 84; 2024: 125; 2025: 123; 2026: 113.

460 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  3. 2024The Platonic Representation Hypothesiscited by 362
  4. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  6. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  10. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  11. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  12. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
38 more
  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  3. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  4. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  6. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  7. 2024Model merging with SVD to tie the Knotscited by 118
  8. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  9. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  10. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  11. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  13. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  14. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  15. 2023Deep Model Fusion: A Surveycited by 89
  16. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  17. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  18. 2023Modular Deep Learningcited by 82
  19. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  20. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  21. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  22. 2024Parameter Competition Balancing for Model Mergingcited by 71
  23. 2023Permutation Equivariant Neural Functionalscited by 69
  24. 2025Latent Collaboration in Multi-Agent Systemscited by 67
  25. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  26. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  27. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  28. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  29. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  30. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  31. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  32. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  33. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  34. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  35. 2022Mechanistic Mode Connectivitycited by 50
  36. 2023Neural Functional Transformerscited by 50
  37. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  38. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50

50 most cited of 460 All 460 (JSON)