# Understanding Mode Connectivity via Parameter Space Symmetry arXiv: https://arxiv.org/abs/2505.23681 · HTML: https://mlsanity.com/paper/2505.23681/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 44 of its 50 references that are in the map. - First posted on arXiv 29 May 2025. ## Builds on (most cited first, 30 of 44) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022) - [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018) - [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019) - [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018) - [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022) - [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020) - [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018) - [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021) - [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019) - [An introduction to Topological Data Analysis: fundamental and practical aspects for data scientists](https://mlsanity.com/paper/1710.04019/) (2017) - [Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness](https://mlsanity.com/paper/2005.00060/) (2020) - [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021) - [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022) - [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016) - [Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics](https://mlsanity.com/paper/2012.04728/) (2020) - [Optimizing Mode Connectivity via Neuron Alignment](https://mlsanity.com/paper/2009.02439/) (2020) - [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021) - [Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets](https://mlsanity.com/paper/1906.06247/) (2019) - [Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape](https://mlsanity.com/paper/1907.02911/) (2019) - [Linear Connectivity Reveals Generalization Strategies](https://mlsanity.com/paper/2205.12411/) (2022) - [On Connected Sublevel Sets in Deep Learning](https://mlsanity.com/paper/1901.07417/) (2019) - [Mechanistic Mode Connectivity](https://mlsanity.com/paper/2211.08422/) (2022) - [Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity](https://mlsanity.com/paper/2307.08286/) (2023) - [Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks](https://mlsanity.com/paper/1912.10095/) (2019) - [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022) - [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022) - [Geometry of Linear Convolutional Networks](https://mlsanity.com/paper/2108.01538/) (2021) ## Built on by (most cited first, 10 of 10) - [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025) - [Fractal basin geometry and the limits to predictability and reproducibility in deep learning](https://mlsanity.com/paper/2510.05606/) (2025) - [Symmetry-Aware Fully-Amortized Optimization with Scale Equivariant Graph Metanetworks](https://mlsanity.com/paper/2510.08300/) (2025) - [Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization](https://mlsanity.com/paper/2511.12601/) (2025) - [From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks](https://mlsanity.com/paper/2602.17596/) (2026) - [Optimizer-Induced Mode Connectivity: From AdamW to Muon](https://mlsanity.com/paper/2605.09991/) (2026) - [Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics](https://mlsanity.com/paper/2605.15459/) (2026) - [Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability](https://mlsanity.com/paper/2606.04754/) (2026) - [Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networks](https://mlsanity.com/paper/2607.16720/) (2026) - [Discovering Symmetries in Neural Network Parameter Spaces](https://mlsanity.com/paper/2609.33527/) (2026) ## Abstract (arXiv) Neural network minima are often connected by curves along which train and test loss remain nearly constant, a phenomenon known as mode connectivity. While this property has enabled applications such as model merging and fine-tuning, its theoretical explanation remains unclear. We propose a new approach to exploring the connectedness of minima using parameter space symmetry. By linking the topology of symmetry groups to that of the minima, we derive the number of connected components of the minima of linear networks and show that skip connections reduce this number. We then examine when mode connectivity and linear mode connectivity hold or fail, using parameter symmetries which account for a significant part of the minimum. Finally, we provide explicit expressions for connecting curves in the minima induced by symmetry. Using the curvature of these curves, we derive conditions under which linear mode connectivity approximately holds. Our findings highlight the role of continuous symmetries in understanding the neural network loss landscape.