# Effect of Depth and Width on Local Minima in Deep Learning

arXiv: https://arxiv.org/abs/1811.08150 · HTML: https://mlsanity.com/paper/1811.08150/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [A Recipe for Global Convergence Guarantee in Deep Neural Networks](https://mlsanity.com/paper/2104.05785/) (2021)
- [Requirements Engineering Framework for Human-centered Artificial Intelligence Software Systems](https://mlsanity.com/paper/2303.02920/) (2023)
- [Doubly Robust Conditional Independence Testing with Generative Neural Networks](https://mlsanity.com/paper/2407.17694/) (2024)
- [Max-Affine Spline Insights Into Deep Network Pruning](https://mlsanity.com/paper/2101.02338/) (2021)
- [Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performance](https://mlsanity.com/paper/2303.03918/) (2023)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticity](https://mlsanity.com/paper/2303.16067/) (2023)
- [Subspace-Configurable Networks](https://mlsanity.com/paper/2305.13536/) (2023)
- [Snapshot Spectral Clustering -- a costless approach to deep clustering ensembles generation](https://mlsanity.com/paper/2307.08591/) (2023)
- [Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss](https://mlsanity.com/paper/2309.06774/) (2023)
- [A Transfer Learning Evaluation of Deep Neural Networks for Image Classification](https://mlsanity.com/paper/2605.11989/) (2026)

## Abstract (arXiv)

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.