# Charting the Topography of the Neural Network Landscape with Thermal-Like Noise

arXiv: https://arxiv.org/abs/2304.01335 · HTML: https://mlsanity.com/paper/2304.01335/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 20 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Built on by (most cited first, 6 of 6)

- [The Local Learning Coefficient: A Singularity-Aware Complexity Measure](https://mlsanity.com/paper/2308.12108/) (2023)
- [Estimating the Local Learning Coefficient at Scale](https://mlsanity.com/paper/2402.03698/) (2024)
- [The Effect of Mini-Batch Noise on the Implicit Bias of Adam](https://mlsanity.com/paper/2602.01642/) (2026)
- [First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models](https://mlsanity.com/paper/2502.04121/) (2025)
- [R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization](https://mlsanity.com/paper/2510.07758/) (2025)
- [Flatness-Aware Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2510.02174/) (2025)

## Abstract (arXiv)

The training of neural networks is a complex, high-dimensional, non-convex and noisy optimization problem whose theoretical understanding is interesting both from an applicative perspective and for fundamental reasons. A core challenge is to understand the geometry and topography of the landscape that guides the optimization. In this work, we employ standard Statistical Mechanics methods, namely, phase-space exploration using Langevin dynamics, to study this landscape for an over-parameterized fully connected network performing a classification task on random data. Analyzing the fluctuation statistics, in analogy to thermal dynamics at a constant temperature, we infer a clear geometric description of the low-loss region. We find that it is a low-dimensional manifold whose dimension can be readily obtained from the fluctuations. Furthermore, this dimension is controlled by the number of data points that reside near the classification decision boundary. Importantly, we find that a quadratic approximation of the loss near the minimum is fundamentally inadequate due to the exponential nature of the decision boundary and the flatness of the low-loss region. This causes the dynamics to sample regions with higher curvature at higher temperatures, while producing quadratic-like statistics at any given temperature. We explain this behavior by a simplified loss model which is analytically tractable and reproduces the observed fluctuation statistics.