mlsanityData to 7 Oct 2026

Paths and Ambient Spaces in Neural Loss Landscapes

arXiv 2503.03382 ↗First posted 5 Mar 2025cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

26 of 43 references.

References · 26

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  4. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  5. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  6. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  9. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  10. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  11. 2019Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMCcited by 90
  12. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
14 more
  1. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  2. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  3. 2013Understanding predictive information criteria for Bayesian modelscited by 51
  4. 2022Mechanistic Mode Connectivitycited by 50
  5. 2022Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometrycited by 27
  6. 2022Random initialisations performing above chance and how to find themcited by 23
  7. 2023Layer-wise Linear Mode Connectivitycited by 22
  8. 2018Using Mode Connectivity for Loss Landscape Analysiscited by 19
  9. 2024What is the long-run distribution of stochastic gradient descent? A large deviations analysiscited by 17
  10. 2025Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networkscited by 10
  11. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  12. 2024Do Deep Neural Network Solutions Form a Star Domain?cited by 6
  13. 2024Bayesian Semi-structured Subspace Inferencecited by 3
  14. 2023Geodesic Mode Connectivitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 2
  2. 2025Aggregation on Learnable Manifolds for Asynchronous Federated Optimizationcited by 1
  3. 2026Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learningcited by 1
  4. 2026Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networkscited by 1
  5. 2025Tubular Riemannian Laplace Approximations for Bayesian Neural Networkscited by 0
  6. 2026On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inferencecited by 0