mlsanityData to 7 Oct 2026

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

arXiv 2606.04754 ↗First posted 3 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 65 of its 76 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

65 of 76 references.

References · 65

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  9. 2016Learning to learn by gradient descent by gradient descentcited by 616
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  12. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
38 more
  1. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  4. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  5. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  6. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  7. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2019Model Fusion via Optimal Transportcited by 217
  10. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  11. 2014Qualitatively characterizing neural network optimization problemscited by 186
  12. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  13. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  14. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  15. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  16. 2020Predicting Neural Network Accuracy from Weightscited by 107
  17. 2019How do infinite width bounded norm networks look in function space?cited by 105
  18. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  19. 2013Dropout Training as Adaptive Regularizationcited by 89
  20. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  21. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  22. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  23. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  24. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  25. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  26. 2023Neural Functional Transformerscited by 50
  27. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  28. 2022Rank Diminishing in Deep Neural Networkscited by 47
  29. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  30. 2023Hidden symmetries of ReLU networkscited by 38
  31. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  32. 2023Transformer Fusion with Optimal Transportcited by 35
  33. 2023Equivariant Deep Weight Space Alignmentcited by 34
  34. 2025Symmetry in Neural Network Parameter Spacescited by 31
  35. 2022Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometrycited by 27
  36. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  37. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  38. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.