# Paths and Ambient Spaces in Neural Loss Landscapes

arXiv: https://arxiv.org/abs/2503.03382 · HTML: https://mlsanity.com/paper/2503.03382/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness](https://mlsanity.com/paper/2005.00060/) (2020)
- [Subspace Inference for Bayesian Deep Learning](https://mlsanity.com/paper/1907.07504/) (2019)
- [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019)
- [Optimizing Mode Connectivity via Neuron Alignment](https://mlsanity.com/paper/2009.02439/) (2020)
- [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021)
- [Large Scale Structure of Neural Network Loss Landscapes](https://mlsanity.com/paper/1906.04724/) (2019)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [Mechanistic Mode Connectivity](https://mlsanity.com/paper/2211.08422/) (2022)
- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Random initialisations performing above chance and how to find them](https://mlsanity.com/paper/2209.07509/) (2022)
- [Layer-wise Linear Mode Connectivity](https://mlsanity.com/paper/2307.06966/) (2023)
- [Using Mode Connectivity for Loss Landscape Analysis](https://mlsanity.com/paper/1806.06977/) (2018)
- [What is the long-run distribution of stochastic gradient descent? A large deviations analysis](https://mlsanity.com/paper/2406.09241/) (2024)
- [Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks](https://mlsanity.com/paper/2502.06335/) (2025)
- [Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity](https://mlsanity.com/paper/2404.06391/) (2024)
- [Do Deep Neural Network Solutions Form a Star Domain?](https://mlsanity.com/paper/2403.07968/) (2024)
- [Bayesian Semi-structured Subspace Inference](https://mlsanity.com/paper/2401.12950/) (2024)
- [Geodesic Mode Connectivity](https://mlsanity.com/paper/2308.12666/) (2023)

## Built on by (most cited first, 6 of 6)

- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [Aggregation on Learnable Manifolds for Asynchronous Federated Optimization](https://mlsanity.com/paper/2503.14396/) (2025)
- [Tubular Riemannian Laplace Approximations for Bayesian Neural Networks](https://mlsanity.com/paper/2512.24381/) (2025)
- [Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning](https://mlsanity.com/paper/2605.21765/) (2026)
- [Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks](https://mlsanity.com/paper/2608.22361/) (2026)
- [On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference](https://mlsanity.com/paper/2605.29580/) (2026)

## Abstract (arXiv)

Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.