# Depth with Nonlinearity Creates No Bad Local Minima in ResNets

arXiv: https://arxiv.org/abs/1810.09038 · HTML: https://mlsanity.com/paper/1810.09038/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2018.

## Built on by (most cited first, 17 of 17)

- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology](https://mlsanity.com/paper/2211.08939/) (2022)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [A Recipe for Global Convergence Guarantee in Deep Neural Networks](https://mlsanity.com/paper/2104.05785/) (2021)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees](https://mlsanity.com/paper/2111.05496/) (2021)
- [Learning distinct features helps, provably](https://mlsanity.com/paper/2106.06012/) (2021)
- [Theoretical Exploration of Flexible Transmitter Model](https://mlsanity.com/paper/2111.06027/) (2021)
- [Learning Unorthogonalized Matrices for Rotation Estimation](https://mlsanity.com/paper/2312.00462/) (2023)
- [System Identification and Control Using Lyapunov-Based Deep Neural Networks without Persistent Excitation: A Concurrent Learning Approach](https://mlsanity.com/paper/2505.10678/) (2025)
- [Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input](https://mlsanity.com/paper/2402.08948/) (2024)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks](https://mlsanity.com/paper/2411.16975/) (2024)
- [Removing spurious minima for planar features by skip connections](https://mlsanity.com/paper/2610.01728/) (2026)

## Abstract (arXiv)

In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations. As a result, this paper provides an affirmative answer to an open question stated in a paper in the conference on Neural Information Processing Systems 2018. This paper advances the optimization theory of deep learning only for ResNets and not for other network architectures.