mlsanityData to 7 Oct 2026

The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions

arXiv 2006.05900 ↗First posted 10 Jun 2020cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

We prove that finding all globally optimal two-layer ReLU neural networks can be performed by solving a convex optimization program with cone constraints. Our analysis is novel, characterizes all optimal solutions, and does not leverage duality-based analysis which was recently used to lift neural network training into convex spaces. Given the set of solutions of our convex optimization program, we show how to construct exactly the entire set of optimal neural networks. We provide a detailed characterization of this optimal set and its invariant transformations. As additional consequences of our convex perspective, (i) we establish that Clarke stationary points found by stochastic gradient descent correspond to the global optimum of a subsampled convex problem (ii) we provide a polynomial-time algorithm for checking if a neural network is a global minimum of the training loss (iii) we provide an explicit construction of a continuous path between any neural network and the global minimum of its sublevel set and (iv) characterize the minimal size of the hidden layer so that the neural network optimization landscape has no spurious valleys. Overall, we provide a rich framework for studying the landscape of neural network training loss through convexity.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 11; 2024: 7; 2025: 2; 2026: 3.

30 papers.

Sort:

  1. 2023Test-Time Training on Video Streamscited by 37
  2. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  3. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  4. 2024The Real Tropical Geometry of Neural Networkscited by 23
  5. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  6. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 16
  7. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  8. 2024Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Timecited by 11
  9. 2023Optimal Sets and Solution Paths of ReLU Networkscited by 9
  10. 2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
  11. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  12. 2023On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrizationcited by 5
18 more
  1. 2021Practical Convex Formulation of Robust One-hidden-layer Neural Network Trainingcited by 4
  2. 2023From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexitycited by 4
  3. 2022A Combinatorial Perspective on the Optimization of Shallow ReLU Networkscited by 3
  4. 2022On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrizationcited by 3
  5. 2023A qualitative difference between gradient flows of convex functions in finite- and infinite-dimensional Hilbert spacescited by 3
  6. 2021Nonasymptotic theory for two-layer neural networks: Beyond the bias-variance trade-offcited by 2
  7. 2022PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networkscited by 2
  8. 2024Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacycited by 2
  9. 2024Wasserstein Distributionally Robust Shallow Convex Neural Networkscited by 2
  10. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  11. 2023Fixing the NTK: From Neural Network Linearizations to Exact Convex Programscited by 1
  12. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  13. 2023Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter ε via Gradient Magnitude Histogram Analysiscited by 0
  14. 2025Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networkscited by 0
  15. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 0
  16. 2026Online Learning on Hidden-Convex Losses via Algorithmic Equivalence: Optimal Regret, Geometric Barrier, and Bandit Feedbackcited by 0
  17. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0
  18. 2026Removing spurious minima for planar features by skip connectionscited by 0