# Local minima in training of neural networks

arXiv: https://arxiv.org/abs/1611.06310 · HTML: https://mlsanity.com/paper/1611.06310/

## Key facts

- Cited by 19 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 19 Nov 2016.

## Built on by (most cited first, 19 of 19)

- [On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks](https://mlsanity.com/paper/2112.09684/) (2021)
- [Special Properties of Gradient Descent with Large Learning Rates](https://mlsanity.com/paper/2205.15142/) (2022)
- [Algebraic Complexity and Neurovariety of Linear Convolutional Networks](https://mlsanity.com/paper/2401.16613/) (2024)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023)
- [On the existence of optimal shallow feedforward networks with ReLU activation](https://mlsanity.com/paper/2303.03950/) (2023)
- [Overparameterization Improves StyleGAN Inversion](https://mlsanity.com/paper/2205.06304/) (2022)
- [Probabilistic Solar Proxy Forecasting with Neural Network Ensembles](https://mlsanity.com/paper/2306.02169/) (2023)
- [Fine-Tuning Hybrid Physics-Informed Neural Networks for Vehicle Dynamics Model Estimation](https://mlsanity.com/paper/2409.19647/) (2024)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks](https://mlsanity.com/paper/2109.09612/) (2021)
- [Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks](https://mlsanity.com/paper/2201.07934/) (2022)
- [Black holes and the loss landscape in machine learning](https://mlsanity.com/paper/2306.14817/) (2023)
- [Learnability of high-dimensional targets by two-parameter models and gradient flow](https://mlsanity.com/paper/2402.17089/) (2024)
- [Weakly Supervised Tuberculosis Localization in Chest X-rays through Knowledge Distillation](https://mlsanity.com/paper/2512.11057/) (2025)
- [Don't stop me now: How Validation Criteria Affect Checkpoint Selection and Early Stopping](https://mlsanity.com/paper/2602.22107/) (2026)
- [Benign Loss Landscapes Can Coexist with Worst-Case Hardness](https://mlsanity.com/paper/2609.13057/) (2026)
- [Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix](https://mlsanity.com/paper/2609.31983/) (2026)

## Abstract (arXiv)

There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima? It is widely believed that training of deep models using gradient methods works so well because the error surface either has no local minima, or if they exist they need to be close in value to the global minimum. It is known that such results hold under very strong assumptions which are not satisfied by real models. In this paper we present examples showing that for such theorem to be true additional assumptions on the data, initialization schemes and/or the model classes have to be made. We look at the particular case of finite size datasets. We demonstrate that in this scenario one can construct counter-examples (datasets or initialization schemes) when the network does become susceptible to bad local minima over the weight space.