# Elimination of All Bad Local Minima in Deep Learning

arXiv: https://arxiv.org/abs/1901.00279 · HTML: https://mlsanity.com/paper/1901.00279/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2019.

## Built on by (most cited first, 10 of 10)

- [Mechanistic Mode Connectivity](https://mlsanity.com/paper/2211.08422/) (2022)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge](https://mlsanity.com/paper/2201.07705/) (2022)
- [Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning](https://mlsanity.com/paper/2203.09137/) (2022)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [On generalization bounds for deep networks based on loss surface implicit regularization](https://mlsanity.com/paper/2201.04545/) (2022)
- [Theoretical Exploration of Flexible Transmitter Model](https://mlsanity.com/paper/2111.06027/) (2021)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning](https://mlsanity.com/paper/2101.02397/) (2021)
- [Blurry Video Compression: A Trade-off between Visual Enhancement and Data Compression](https://mlsanity.com/paper/2311.04430/) (2023)

## Abstract (arXiv)

In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function, under practical assumptions. At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network. The effects of the added neurons are proven to automatically vanish at every local minimum. Moreover, we provide a novel theoretical characterization of a failure mode of eliminating suboptimal local minima via an additional theorem and several examples. This paper also introduces a novel proof technique based on the perturbable gradient basis (PGB) necessary condition of local minima, which provides new insight into the elimination of local minima and is applicable to analyze various models and transformations of objective functions beyond the elimination of local minima.