mlsanityData to 7 Oct 2026

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

arXiv 2110.06296 ↗First posted 12 Oct 2021cs.LGcited by 285

Not on the map. Why?

Abstract (arXiv)

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

34 of 45 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  12. 2017Exploring Generalization in Deep Learningcited by 595
22 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  3. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  4. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  5. 2020What is being transferred in transfer learning?cited by 365
  6. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  7. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  8. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  9. 2019Model Fusion via Optimal Transportcited by 217
  10. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  11. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  12. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  13. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  14. 2021Learning Neural Network Subspacescited by 82
  15. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  16. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
  17. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  18. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  19. 2020Towards Learning Convolutions from Scratchcited by 46
  20. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  21. 2018Multi-Task Zipping via Layer-wise Neuron Sharingcited by 20
  22. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 62; 2024: 75; 2025: 58; 2026: 65.

285 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  8. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  9. 2022Patching open-vocabulary models by interpolating weightscited by 187
  10. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  11. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  12. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
38 more
  1. 2024Model merging with SVD to tie the Knotscited by 118
  2. 2022Fusing finetuned models for better pretrainingcited by 110
  3. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  4. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  5. 2023Deep Model Fusion: A Surveycited by 89
  6. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  7. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  8. 2023Permutation Equivariant Neural Functionalscited by 69
  9. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  10. 2023Deep Learning on Implicit Neural Representations of Shapescited by 62
  11. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  12. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  13. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  14. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  15. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  16. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  17. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  18. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  19. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  20. 2024What Matters for Model Merging at Scale?cited by 52
  21. 2022Mechanistic Mode Connectivitycited by 50
  22. 2023Neural Functional Transformerscited by 50
  23. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  24. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  25. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  26. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  27. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  28. 2023Hidden symmetries of ReLU networkscited by 38
  29. 2024Model Extrapolation Expedites Alignmentcited by 37
  30. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  31. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 36
  32. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  33. 2023Transformer Fusion with Optimal Transportcited by 35
  34. 2023Equivariant Deep Weight Space Alignmentcited by 34
  35. 2024Training-Free Pretrained Model Mergingcited by 32
  36. 2025Symmetry in Neural Network Parameter Spacescited by 31
  37. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  38. 2022Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Taskscited by 29

50 most cited of 285 All 285 (JSON)