# Adding One Neuron Can Eliminate All Bad Local Minima

arXiv: https://arxiv.org/abs/1805.08671 · HTML: https://mlsanity.com/paper/1805.08671/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2018.

## Built on by (most cited first, 21 of 21)

- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [Global Optimality in Bivariate Gradient-based DAG Learning](https://mlsanity.com/paper/2306.17378/) (2023)
- [Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity](https://mlsanity.com/paper/2404.06391/) (2024)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [Achieving Small Test Error in Mildly Overparameterized Neural Networks](https://mlsanity.com/paper/2104.11895/) (2021)
- [Theoretical Exploration of Flexible Transmitter Model](https://mlsanity.com/paper/2111.06027/) (2021)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity](https://mlsanity.com/paper/2510.08023/) (2025)
- [A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning](https://mlsanity.com/paper/2101.02397/) (2021)
- [WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points](https://mlsanity.com/paper/2102.07541/) (2021)
- [Training Deep Neural Networks via Branch-and-Bound](https://mlsanity.com/paper/2104.01730/) (2021)
- [Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer](https://mlsanity.com/paper/2201.02761/) (2022)
- [Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provably](https://mlsanity.com/paper/2202.03535/) (2022)
- [Blurry Video Compression: A Trade-off between Visual Enhancement and Data Compression](https://mlsanity.com/paper/2311.04430/) (2023)
- [Analysis of the rate of convergence of an over-parametrized convolutional neural network image classifier learned by gradient descent](https://mlsanity.com/paper/2405.07619/) (2024)
- [Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guarantees](https://mlsanity.com/paper/2609.16485/) (2026)
- [Removing spurious minima for planar features by skip connections](https://mlsanity.com/paper/2610.01728/) (2026)

## Abstract (arXiv)

One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.