mlsanityData to 7 Oct 2026

On Connected Sublevel Sets in Deep Learning

arXiv 1901.07417 ↗First posted 22 Jan 2019cs.LG stat.MLcited by 51

Not on the map. Why?

Abstract (arXiv)

This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 7; 2023: 8; 2024: 6; 2025: 11; 2026: 5.

51 papers.

Sort:

  1. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  2. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  3. 2023Deep Model Fusion: A Surveycited by 89
  4. 2022Mechanistic Mode Connectivitycited by 50
  5. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  6. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  7. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  8. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  9. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  10. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  11. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 25
  12. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
38 more
  1. 2022Neural Tangent Kernel: A Surveycited by 22
  2. 2023DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networkscited by 20
  3. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 16
  4. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  5. 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
  6. 2025Rethinking Spiking Neural Networks from an Ensemble Learning Perspectivecited by 12
  7. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  8. 2021When Are Solutions Connected in Deep Networks?cited by 10
  9. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10
  10. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  11. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  12. 2023Optimal Sets and Solution Paths of ReLU Networkscited by 9
  13. 2021A Note on Connectivity of Sublevel Sets in Deep Learningcited by 8
  14. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 8
  15. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  16. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  17. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  18. 2022Flatter, faster: scaling momentum for optimal speedup of SGDcited by 5
  19. 2023Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theoremscited by 3
  20. 2023Mode Connectivity in Auction Designcited by 3
  21. 2025A mathematical theory for understanding when abstract representations emerge in neural networkscited by 3
  22. 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
  23. 2025Analyzing the Role of Permutation Invariance in Linear Mode Connectivitycited by 2
  24. 2025Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivitycited by 2
  25. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 2
  26. 2021Proactive DP: A Multple Target Optimization Framework for DP-SGDcited by 1
  27. 2021Structured Directional Pruning via Perturbation Orthogonal Projectioncited by 1
  28. 2022Deep learning, stochastic gradient descent and diffusion mapscited by 1
  29. 2023Evolutionary Algorithms in the Light of SGD: Limit Equivalence, Minima Flatness, and Transfer Learningcited by 1
  30. 2025SE-Merging: A Self-Enhanced Approach for Dynamic Model Mergingcited by 1
  31. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  32. 2025Find A Winning Sign: Sign Is All We Need to Win the Lotterycited by 0
  33. 2025In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoodscited by 0
  34. 2025Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Liftingcited by 0
  35. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 0
  36. 2026From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networkscited by 0
  37. 2026The Role of Symmetry in Optimizing Overparameterized Networkscited by 0
  38. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0

50 most cited of 51 All 51 (JSON)