mlsanityData to 7 Oct 2026

Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity

arXiv 2404.06391 ↗First posted 9 Apr 2024cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

One of the most intriguing findings in the structure of neural network landscape is the phenomenon of mode connectivity: For two typical global minima, there exists a path connecting them without barrier. This concept of mode connectivity has played a crucial role in understanding important phenomena in deep learning. In this paper, we conduct a fine-grained analysis of this connectivity phenomenon. First, we demonstrate that in the overparameterized case, the connecting path can be as simple as a two-piece linear path, and the path length can be nearly equal to the Euclidean distance. This finding suggests that the landscape should be nearly convex in a certain sense. Second, we uncover a surprising star-shaped connectivity: For a finite number of typical minima, there exists a center on minima manifold that connects all of them simultaneously via linear paths. These results are provably valid for linear networks and two-layer ReLU networks under a teacher-student setup, and are empirically supported by models trained on MNIST and CIFAR-10.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

42 of 51 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  5. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  10. 2017Three Factors Influencing Minima in SGDcited by 270
  11. 2016Deep Learning without Poor Local Minimacited by 233
  12. 2014Qualitatively characterizing neural network optimization problemscited by 186
30 more
  1. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  2. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  3. 2016Matrix Completion has No Spurious Local Minimumcited by 124
  4. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  5. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  6. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  7. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  8. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  9. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  10. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  11. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  12. 2018The loss landscape of overparameterized neural networkscited by 38
  13. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  14. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  15. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  16. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  17. 2017Depth Creates No Bad Local Minimacited by 29
  18. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 29
  19. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  20. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  21. 2018Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitionscited by 23
  22. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 22
  23. 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
  24. 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
  25. 2020Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidancecited by 11
  26. 2021When Are Solutions Connected in Deep Networks?cited by 10
  27. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10
  28. 2018Non-attracting Regions of Local Minima in Deep and Wide Neural Networkscited by 8
  29. 2023The star-shaped space of solutions of the spherical negative perceptroncited by 8
  30. 2020On the Landscape of One-hidden-layer Sparse Networks and Beyondcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  2. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  3. 2025Paths and Ambient Spaces in Neural Loss Landscapescited by 6
  4. 2025Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oraclescited by 3
  5. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  6. 2025Analyzing the Role of Permutation Invariance in Linear Mode Connectivitycited by 2
  7. 2025Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 0
  8. 2026A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networkscited by 0
  9. 2026Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 0