mlsanityData to 13 Oct 2026

Disentangling Linear Mode-Connectivity

arXiv 2312.09832 ↗First posted 15 Dec 2023cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes. While empirically well established, it unfortunately still lacks a proper theoretical understanding. Even worse, although empirical data points are abound, a systematic study of when networks exhibit LMC is largely missing in the literature. In this work we aim to close this gap. We explore how LMC is affected by three factors: (1) architecture (sparsity, weight-sharing), (2) training strategy (optimization setup) as well as (3) the underlying dataset. We place particular emphasis on minimal but non-trivial settings, removing as much unnecessary complexity as possible. We believe that our insights can guide future theoretical works on uncovering the inner workings of LMC.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 18 of its 19 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

18 of 19 references.

References · 18

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 631
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 586
  4. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 463
  5. 2020What is being transferred in transfer learning?cited by 366
  6. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 329
  7. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 287
  9. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  10. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  11. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  12. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
6 more
  1. 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
  2. 2020Towards Learning Convolutions from Scratchcited by 46
  3. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  4. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 28
  5. 2022Random initialisations performing above chance and how to find themcited by 23
  6. 2023Towards Democratizing Joint-Embedding Self-Supervised Learningcited by 18

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 3.

8 papers.

  1. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  2. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  3. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  4. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  5. 2024Do Deep Neural Network Solutions Form a Star Domain?cited by 6
  6. 2024Unveiling the Dynamics of Information Interplay in Supervised Learningcited by 3
  7. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Trainingcited by 1
  8. 2025Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifierscited by 1