mlsanityData to 14 Oct 2026

Layer-wise Linear Mode Connectivity

arXiv 2307.06966 ↗First posted 13 Jul 2023cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are averaged at the end of training, this can only lead to a good performing model if the loss surface of interest is very particular, i.e., the loss in the midpoint between the two models needs to be sufficiently low. This is impossible to guarantee for the non-convex losses of state-of-the-art networks. For averaging models trained on vastly different datasets, it was proposed to average only the parameters of particular layers or combinations of layers, resulting in better performing models. To get a better understanding of the effect of layer-wise averaging, we analyse the performance of the models that result from averaging single layers, or groups of layers. Based on our empirical and theoretical investigation, we introduce a novel notion of the layer-wise linear connectivity, and show that deep networks do not have layer-wise barriers between them.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

46 of 58 references.

References · 46

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  6. 2022Editing Models with Task Arithmeticcited by 1,120
  7. 2018Robustness May Be at Odds with Accuracycited by 880
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  10. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  11. 2020Federated Learning with Matched Averagingcited by 564
  12. 2019Federated Learning with Personalization Layerscited by 541
34 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  3. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  4. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  5. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  6. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  7. 2019Model Fusion via Optimal Transportcited by 219
  8. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  9. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  10. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  11. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  12. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  13. 2022Federated Learning with Partial Model Personalizationcited by 114
  14. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 96
  15. 2019Are All Layers Created Equal?cited by 94
  16. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  17. 2020Relative Flatness and Generalizationcited by 88
  18. 2021Learning Neural Network Subspacescited by 82
  19. 2018A Walk with SGDcited by 78
  20. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  21. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  22. 2022Mechanistic Mode Connectivitycited by 50
  23. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  24. 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
  25. 2022How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularizationcited by 40
  26. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  27. 2022CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNetcited by 35
  28. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  29. 2018Efficient Decentralized Deep Learning by Dynamic Model Averagingcited by 30
  30. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 28
  31. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  32. 2022Revisiting adapters with adversarial trainingcited by 18
  33. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  34. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 8; 2026: 3.

22 papers.

Sort:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2023Deep Model Fusion: A Surveycited by 91
  3. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  4. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  5. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 26
  6. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 25
  7. 2025Generalized Linear Mode Connectivity for Transformerscited by 16
  8. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  9. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 9
  10. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  11. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  12. 2024Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchorscited by 7
10 more
  1. 2024Landscaping Linear Mode Connectivitycited by 7
  2. 2025Paths and Ambient Spaces in Neural Loss Landscapescited by 6
  3. 2024Vanishing Feature: Diagnosing Model Merging and Beyondcited by 5
  4. 2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
  5. 2024The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspectivecited by 4
  6. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3
  7. 2026Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformerscited by 2
  8. 2024CopRA: A Progressive LoRA Training Strategycited by 1
  9. 2026The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imagingcited by 0
  10. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0