mlsanityData to 14 Oct 2026

Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks

arXiv 2210.06671 ↗First posted 13 Oct 2022cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Based on the concepts of Wasserstein barycenter (WB) and Gromov-Wasserstein barycenter (GWB), we propose a unified mathematical framework for neural network (NN) model fusion and utilize it to reveal new insights about the linear mode connectivity of SGD solutions. In our framework, the fusion occurs in a layer-wise manner and builds on an interpretation of a node in a network as a function of the layer preceding it. The versatility of our mathematical framework allows us to talk about model fusion and linear mode connectivity for a broad class of NNs, including fully connected NN, CNN, ResNet, RNN, and LSTM, in each case exploiting the specific structure of the network architecture. We present extensive numerical experiments to: 1) illustrate the strengths of our approach in relation to other model fusion methodologies and 2) from a certain perspective, provide new empirical evidence for recent conjectures which say that two local minima found by gradient-based methods end up lying on the same basin of the loss landscape after a proper permutation of weights is applied to one of the models.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

39 of 63 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  11. 2018Computational Optimal Transportcited by 780
  12. 2017Exploring Generalization in Deep Learningcited by 600
27 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  2. 2020Federated Learning with Matched Averagingcited by 564
  3. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  5. 2015Optimal Transport for Domain Adaptationcited by 463
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  7. 2020What is being transferred in transfer learning?cited by 368
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  9. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  10. 2019Model Fusion via Optimal Transportcited by 219
  11. 2013Fast Computation of Wasserstein Barycenterscited by 216
  12. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  13. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  14. 2019Gromov-Wasserstein Learning for Graph Matching and Node Embeddingcited by 126
  15. 2019Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matchingcited by 103
  16. 2018Optimal Transport for Multi-source Domain Adaptation under Target Shiftcited by 68
  17. 2017Multilevel Clustering via Wasserstein Meanscited by 49
  18. 2015Scalable Bayes via Barycenter in Wasserstein Spacecited by 44
  19. 2021Fed2: Feature-Aligned Federated Learningcited by 42
  20. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  21. 2014Continuum limit of total variation on point cloudscited by 24
  22. 2022Federated Learning with Position-Aware Neuronscited by 24
  23. 2022Random initialisations performing above chance and how to find themcited by 23
  24. 2015Discrete Wasserstein Barycenters: Optimal Transport for Discrete Datacited by 20
  25. 2016A Transportation Lp Distance for Signal Analysiscited by 11
  26. 2019Wasserstein Barycenter Model Ensemblingcited by 10
  27. 2020Model Fusion with Kullback--Leibler Divergencecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 5; 2025: 3; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023Deep Model Fusion: A Surveycited by 91
  2. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  3. 2023Transformer Fusion with Optimal Transportcited by 36
  4. 2023Equivariant Deep Weight Space Alignmentcited by 35
  5. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 25
  6. 2024FedFisher: Leveraging Fisher Information for One-Shot Federated Learningcited by 15
  7. 2024Simultaneous linear connectivity of neural networks modulo permutationcited by 10
  8. 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
  9. 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
  10. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  11. 2025Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenterscited by 4
  12. 2026Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivitycited by 3
3 more
  1. 2024A Lipschitz spaces view of infinitely wide shallow neural networkscited by 2
  2. 2025Robustness and Regularization in Hierarchical Re-Basincited by 1
  3. 2026Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregationcited by 1