mlsanityData to 14 Oct 2026

Optimal Sets and Solution Paths of ReLU Networks

arXiv 2306.00119 ↗First posted 31 May 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We develop an analytical framework to characterize the set of optimal ReLU neural networks by reformulating the non-convex training problem as a convex program. We show that the global optima of the convex parameterization are given by a polyhedral set and then extend this characterization to the optimal set of the non-convex training objective. Since all stationary points of the ReLU training problem can be represented as optima of sub-sampled convex programs, our work provides a general expression for all critical points of the non-convex objective. We then leverage our results to provide an optimal pruning algorithm for computing minimal networks, establish conditions for the regularization path of ReLU networks to be continuous, and develop sensitivity results for minimal ReLU networks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 18 of its 47 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

18 of 47 references.

References · 18

  1. 2021Convex optimizationcited by 2,515
  2. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  4. 2020What is the State of Neural Network Pruning?cited by 583
  5. 2019Differentiable Convex Optimization Layerscited by 465
  6. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  8. 2017Implicit Regularization in Matrix Factorizationcited by 322
  9. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  10. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  11. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  12. 2019On Connected Sublevel Sets in Deep Learningcited by 51
6 more
  1. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 34
  2. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  3. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  4. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  5. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  6. 2019Decoupling Gating from Linearitycited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 5.

10 papers.

  1. 2024The Real Tropical Geometry of Neural Networkscited by 24
  2. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  3. 2024Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimizationcited by 1
  4. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 1
  5. 2026Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspectivecited by 1
  6. 2025Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Networkcited by 0
  7. 2026From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networkscited by 0
  8. 2026Optimizer-Induced Mode Connectivity: From AdamW to Muoncited by 0
  9. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0
  10. 2026Dimension-Free Decentralized Nonsmooth Nonconvex Stochastic Optimizationcited by 0