# Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity arXiv: https://arxiv.org/abs/2404.06391 ยท HTML: https://mlsanity.com/paper/2404.06391/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 42 of its 51 references that are in the map. - First posted on arXiv 9 Apr 2024. ## Builds on (most cited first, 30 of 42) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016) - [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019) - [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018) - [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022) - [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018) - [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021) - [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017) - [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017) - [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016) - [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014) - [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020) - [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021) - [Matrix Completion has No Spurious Local Minimum](https://mlsanity.com/paper/1605.07272/) (2016) - [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017) - [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016) - [Asymmetric Valleys: Beyond Sharp and Flat Local Minima](https://mlsanity.com/paper/1902.00744/) (2019) - [Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes](https://mlsanity.com/paper/1706.10239/) (2017) - [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021) - [The alignment property of SGD noise and how it helps select flat minima: A stability analysis](https://mlsanity.com/paper/2207.02628/) (2022) - [Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets](https://mlsanity.com/paper/1906.06247/) (2019) - [On Linear Stability of SGD and Input-Smoothness of Neural Networks](https://mlsanity.com/paper/2105.13462/) (2021) - [The loss landscape of overparameterized neural networks](https://mlsanity.com/paper/1804.10200/) (2018) - [On the loss landscape of a class of deep neural networks with no bad local valleys](https://mlsanity.com/paper/1809.10749/) (2018) - [Embedding Principle of Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2105.14573/) (2021) - [Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks](https://mlsanity.com/paper/1912.10095/) (2019) - [No bad local minima: Data independent training error guarantees for multilayer neural networks](https://mlsanity.com/paper/1605.08361/) (2016) - [Semi-flat minima and saddle points by embedding neural networks to overparameterization](https://mlsanity.com/paper/1906.04868/) (2019) - [Depth Creates No Bad Local Minima](https://mlsanity.com/paper/1702.08580/) (2017) ## Built on by (most cited first, 9 of 9) - [MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning](https://mlsanity.com/paper/2407.20999/) (2024) - [The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions](https://mlsanity.com/paper/2506.13234/) (2025) - [Paths and Ambient Spaces in Neural Loss Landscapes](https://mlsanity.com/paper/2503.03382/) (2025) - [Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles](https://mlsanity.com/paper/2505.02281/) (2025) - [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025) - [Analyzing the Role of Permutation Invariance in Linear Mode Connectivity](https://mlsanity.com/paper/2503.06001/) (2025) - [Understanding Machine Unlearning Through the Lens of Mode Connectivity](https://mlsanity.com/paper/2504.06407/) (2025) - [A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks](https://mlsanity.com/paper/2606.04327/) (2026) - [Understanding Machine Unlearning Through the Lens of Mode Connectivity](https://mlsanity.com/paper/2607.23970/) (2026) ## Abstract (arXiv) One of the most intriguing findings in the structure of neural network landscape is the phenomenon of mode connectivity: For two typical global minima, there exists a path connecting them without barrier. This concept of mode connectivity has played a crucial role in understanding important phenomena in deep learning. In this paper, we conduct a fine-grained analysis of this connectivity phenomenon. First, we demonstrate that in the overparameterized case, the connecting path can be as simple as a two-piece linear path, and the path length can be nearly equal to the Euclidean distance. This finding suggests that the landscape should be nearly convex in a certain sense. Second, we uncover a surprising star-shaped connectivity: For a finite number of typical minima, there exists a center on minima manifold that connects all of them simultaneously via linear paths. These results are provably valid for linear networks and two-layer ReLU networks under a teacher-student setup, and are empirically supported by models trained on MNIST and CIFAR-10.