# Explorations on high dimensional landscapes

arXiv: https://arxiv.org/abs/1412.6615 · HTML: https://mlsanity.com/paper/1412.6615/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2014.

## Built on by (most cited first, 8 of 8)

- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [Exponentially Many Local Minima in Quantum Neural Networks](https://mlsanity.com/paper/2110.02479/) (2021)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Appearance of Random Matrix Theory in Deep Learning](https://mlsanity.com/paper/2102.06740/) (2021)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)
- [A survey of deep learning optimizers -- first and second order methods](https://mlsanity.com/paper/2211.15596/) (2022)
- [A Local Optima Network Analysis of the Feedforward Neural Architecture Space](https://mlsanity.com/paper/2206.06903/) (2022)
- [CopRA: A Progressive LoRA Training Strategy](https://mlsanity.com/paper/2410.22911/) (2024)

## Abstract (arXiv)

Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.