# On Connected Sublevel Sets in Deep Learning

arXiv: https://arxiv.org/abs/1901.07417 · HTML: https://mlsanity.com/paper/1901.07417/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2019.

## Built on by (most cited first, 30 of 51)

- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Mechanistic Mode Connectivity](https://mlsanity.com/paper/2211.08422/) (2022)
- [Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity](https://mlsanity.com/paper/2307.08286/) (2023)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes](https://mlsanity.com/paper/2102.09385/) (2021)
- [Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes](https://mlsanity.com/paper/2104.11044/) (2021)
- [On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm](https://mlsanity.com/paper/2402.03660/) (2024)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Neural Tangent Kernel: A Survey](https://mlsanity.com/paper/2208.13614/) (2022)
- [DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks](https://mlsanity.com/paper/2302.14685/) (2023)
- [Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape](https://mlsanity.com/paper/2305.19510/) (2023)
- [Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization](https://mlsanity.com/paper/2102.12430/) (2021)
- [A theory of representation learning gives a deep generalisation of kernel methods](https://mlsanity.com/paper/2108.13097/) (2021)
- [Parameter Symmetry Potentially Unifies Deep Learning Theory](https://mlsanity.com/paper/2502.05300/) (2025)
- [Rethinking Spiking Neural Networks from an Ensemble Learning Perspective](https://mlsanity.com/paper/2502.14218/) (2025)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networks](https://mlsanity.com/paper/2210.01019/) (2022)
- [Optimal Sets and Solution Paths of ReLU Networks](https://mlsanity.com/paper/2306.00119/) (2023)
- [Understanding Mode Connectivity via Parameter Space Symmetry](https://mlsanity.com/paper/2505.23681/) (2025)
- [On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walks](https://mlsanity.com/paper/2209.14501/) (2022)
- [Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods](https://mlsanity.com/paper/2402.04051/) (2024)
- [A Note on Connectivity of Sublevel Sets in Deep Learning](https://mlsanity.com/paper/2101.08576/) (2021)
- [How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD](https://mlsanity.com/paper/2406.11110/) (2024)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [Flatter, faster: scaling momentum for optimal speedup of SGD](https://mlsanity.com/paper/2210.16400/) (2022)

## Abstract (arXiv)

This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.