# Understanding the Loss Surface of Neural Networks for Binary Classification

arXiv: https://arxiv.org/abs/1803.00909 · HTML: https://mlsanity.com/paper/1803.00909/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2018.

## Built on by (most cited first, 27 of 27)

- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity](https://mlsanity.com/paper/2307.08286/) (2023)
- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm](https://mlsanity.com/paper/2402.03660/) (2024)
- [Noise-robust Graph Learning by Estimating and Leveraging Pairwise Interactions](https://mlsanity.com/paper/2106.07451/) (2021)
- [Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks](https://mlsanity.com/paper/2302.02922/) (2023)
- [Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization](https://mlsanity.com/paper/2102.12430/) (2021)
- [Understanding the Learning Dynamics of Alignment with Human Feedback](https://mlsanity.com/paper/2403.18742/) (2024)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [On the Learnability of Out-of-distribution Detection](https://mlsanity.com/paper/2404.04865/) (2024)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees](https://mlsanity.com/paper/2111.05496/) (2021)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/2101.04243/) (2021)
- [Achieving Small Test Error in Mildly Overparameterized Neural Networks](https://mlsanity.com/paper/2104.11895/) (2021)
- [Theoretical Exploration of Flexible Transmitter Model](https://mlsanity.com/paper/2111.06027/) (2021)
- [Empirical Loss Landscape Analysis of Neural Network Activation Functions](https://mlsanity.com/paper/2306.16090/) (2023)
- [Analyzing the Role of Permutation Invariance in Linear Mode Connectivity](https://mlsanity.com/paper/2503.06001/) (2025)
- [WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points](https://mlsanity.com/paper/2102.07541/) (2021)
- [Deep learning, stochastic gradient descent and diffusion maps](https://mlsanity.com/paper/2204.01365/) (2022)
- [Xtreme Margin: A Tunable Loss Function for Binary Classification Problems](https://mlsanity.com/paper/2211.00176/) (2022)
- [SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging](https://mlsanity.com/paper/2506.18135/) (2025)
- [Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer](https://mlsanity.com/paper/2201.02761/) (2022)
- [Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks](https://mlsanity.com/paper/2201.07934/) (2022)

## Abstract (arXiv)

It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performance and generalization performance. Here we focus on the training performance of single-layered neural networks for binary classification, and provide conditions under which the training error is zero at all local minima of a smooth hinge loss function. Our conditions are roughly in the following form: the neurons have to be strictly convex and the surrogate loss function should be a smooth version of hinge loss. We also provide counterexamples to show that when the loss function is replaced with quadratic loss or logistic loss, the result may not hold.