# The Goldilocks zone: Towards better understanding of neural network loss landscapes

arXiv: https://arxiv.org/abs/1807.02581 · HTML: https://mlsanity.com/paper/1807.02581/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2018.

## Built on by (most cited first, 30 of 32)

- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes](https://mlsanity.com/paper/2104.11044/) (2021)
- [How many degrees of freedom do we need to train deep networks: a loss landscape perspective](https://mlsanity.com/paper/2107.05802/) (2021)
- [Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition](https://mlsanity.com/paper/2408.08944/) (2024)
- [On the Power-Law Hessian Spectrums in Deep Learning](https://mlsanity.com/paper/2201.13011/) (2022)
- [Latent State Models of Training Dynamics](https://mlsanity.com/paper/2308.09543/) (2023)
- [Deep Grokking: Would Deep Neural Networks Generalize Better?](https://mlsanity.com/paper/2405.19454/) (2024)
- [On the Overlooked Structure of Stochastic Gradients](https://mlsanity.com/paper/2212.02083/) (2022)
- [A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2310.08287/) (2023)
- [Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization](https://mlsanity.com/paper/2201.11137/) (2022)
- [What Can Grokking Teach Us About Learning Under Nonstationarity?](https://mlsanity.com/paper/2507.20057/) (2025)
- [What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries](https://mlsanity.com/paper/2210.05546/) (2022)
- [LTAU-FF: Loss Trajectory Analysis for Uncertainty in Atomistic Force Fields](https://mlsanity.com/paper/2402.00853/) (2024)
- [Deconstructing the Goldilocks Zone of Neural Network Initialization](https://mlsanity.com/paper/2402.03579/) (2024)
- [High-entropy Advantage in Neural Networks' Generalizability](https://mlsanity.com/paper/2503.13145/) (2025)
- [A survey of deep learning optimizers -- first and second order methods](https://mlsanity.com/paper/2211.15596/) (2022)
- [Acceleration of Grokking in Learning Arithmetic Operations via Kolmogorov-Arnold Representation](https://mlsanity.com/paper/2405.16658/) (2024)
- [FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks](https://mlsanity.com/paper/2204.04511/) (2022)
- [Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2405.12493/) (2024)
- [What Scales in Cross-Entropy Scaling Law?](https://mlsanity.com/paper/2510.04067/) (2025)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysis](https://mlsanity.com/paper/2602.07135/) (2026)
- [Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?](https://mlsanity.com/paper/2606.08578/) (2026)
- [Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks](https://mlsanity.com/paper/2201.07934/) (2022)
- [A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minima](https://mlsanity.com/paper/2203.10973/) (2022)
- [Hyper-Learning for Gradient-Based Batch Size Adaptation](https://mlsanity.com/paper/2205.08231/) (2022)
- [Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation](https://mlsanity.com/paper/2403.05532/) (2024)

## Abstract (arXiv)

We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, H, of the loss function on these hypersurfaces, we observe 1) an unusual excess of the number of positive eigenvalues of H, and 2) a large value of Tr(H) / ||H|| at a well defined range of configuration space radii, corresponding to a thick, hollow, spherical shell we refer to as the textitGoldilocks zone. We observe this effect for fully-connected neural networks over a range of network widths and depths on MNIST and CIFAR-10 datasets with the ReLU and tanh non-linearities, and a similar effect for convolutional networks. Using our observations, we demonstrate a close connection between the Goldilocks zone, measures of local convexity/prevalence of positive curvature, and the suitability of a network initialization. We show that the high and stable accuracy reached when optimizing on random, low-dimensional hypersurfaces is directly related to the overlap between the hypersurface and the Goldilocks zone, and as a corollary demonstrate that the notion of intrinsic dimension is initialization-dependent. We note that common initialization techniques initialize neural networks in this particular region of unusually high convexity/prevalence of positive curvature, and offer a geometric intuition for their success. Furthermore, we demonstrate that initializing a neural network at a number of points and selecting for high measures of local convexity such as Tr(H) / ||H||, number of positive eigenvalues of H, or low initial loss, leads to statistically significantly faster training on MNIST. Based on our observations, we hypothesize that the Goldilocks zone contains an unusually high density of suitable initialization configurations.