mlsanityData to 7 Oct 2026

Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networks

arXiv 2607.16720 ↗First posted 18 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Understanding deep neural networks remains a central challenge in machine learning. In particular, the theoretical properties of even two-layer ReLU networks, especially in the presence of weight decay, remain poorly understood. To this end, we derive a sufficient condition on the hyperparameter settings under which the global minima collapse to the zero solution. Interestingly, our experiments reveal that using AdamW as an optimizer prevents the collapse of the learned parameters, whereas using SGD does not, which may help explain the success of AdamW in deep learning training. In addition, when restricting the input dimension to one, we derive an analytical solution for the globally optimal parameter sets of two-layer ReLU networks and show that ℓ2-regularization has a width-invariant effect on connectivity, but its dimensionality-reducing effect becomes stronger as the network width increases. These results provide insight into how width-dependent hyperparameters influence the geometry of regularized loss landscapes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 18 Jul 2026.

Builds on

31 of 42 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  8. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  9. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  10. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  11. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  12. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
19 more
  1. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  2. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  3. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  4. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  5. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  6. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  7. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  8. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  9. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  10. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  11. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 16
  12. 2024Exploring the loss landscape of regularized neural networks via convex dualitycited by 11
  13. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  14. 2023Optimal Sets and Solution Paths of ReLU Networkscited by 9
  15. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  16. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 6
  17. 2025Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds Itcited by 5
  18. 2024Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalizationcited by 4
  19. 2024A generalized neural tangent kernel for surrogate gradient learningcited by 2

Built on by

Not cited yet.