# Using Mode Connectivity for Loss Landscape Analysis

arXiv: https://arxiv.org/abs/1806.06977 · HTML: https://mlsanity.com/paper/1806.06977/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2018.

## Built on by (most cited first, 20 of 20)

- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Proving Linear Mode Connectivity of Neural Networks via Optimal Transport](https://mlsanity.com/paper/2310.19103/) (2023)
- [Merging Vision Transformers from Different Tasks and Domains](https://mlsanity.com/paper/2312.16240/) (2023)
- [Understanding Mode Connectivity via Parameter Space Symmetry](https://mlsanity.com/paper/2505.23681/) (2025)
- [Lookaround Optimizer: k steps around, 1 step average](https://mlsanity.com/paper/2306.07684/) (2023)
- [On Linear Mode Connectivity of Mixture-of-Experts Architectures](https://mlsanity.com/paper/2509.11348/) (2025)
- [Do Deep Neural Network Solutions Form a Star Domain?](https://mlsanity.com/paper/2403.07968/) (2024)
- [Paths and Ambient Spaces in Neural Loss Landscapes](https://mlsanity.com/paper/2503.03382/) (2025)
- [Inductive biases in deep learning models for weather prediction](https://mlsanity.com/paper/2304.04664/) (2023)
- [Unveiling Mode Connectivity in Graph Neural Networks](https://mlsanity.com/paper/2502.12608/) (2025)
- [Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity](https://mlsanity.com/paper/2606.17830/) (2026)
- [Consistent Explanations in the Face of Model Indeterminacy via Ensembling](https://mlsanity.com/paper/2306.06193/) (2023)
- [Circumventing Backdoor Space via Weight Symmetry](https://mlsanity.com/paper/2506.07467/) (2025)
- [Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training](https://mlsanity.com/paper/2605.26484/) (2026)
- [Geometrically Guided Integrated Gradients](https://mlsanity.com/paper/2206.05903/) (2022)
- [Universally Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergence](https://mlsanity.com/paper/2407.14710/) (2024)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [Exploring End-to-end Differentiable Neural Charged Particle Tracking -- A Loss Landscape Perspective](https://mlsanity.com/paper/2407.13420/) (2024)

## Abstract (arXiv)

Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trained or initialized differently. We find that the procedure is resilient to such changes. Given this finding, we propose using the framework for analyzing loss surfaces and training trajectories more generally, and in this direction, study SGD with cosine annealing and restarts (SGDR). We report that while SGDR moves over barriers in its trajectory, propositions claiming that it converges to and escapes from multiple local minima are not substantiated by our empirical results.