mlsanityData to 13 Oct 2026

Convex Geometry and Duality of Over-parameterized Neural Networks

arXiv 2002.11219 ↗First posted 25 Feb 2020cs.LG stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

We develop a convex analytic approach to analyze finite width two-layer ReLU networks. We first prove that an optimal solution to the regularized training problem can be characterized as extreme points of a convex set, where simple solutions are encouraged via its convex geometrical properties. We then leverage this characterization to show that an optimal set of parameters yield linear spline interpolation for regression problems involving one dimensional or rank-one data. We also characterize the classification decision regions in terms of a kernel matrix and minimum ℓ1-norm solutions. This is in contrast to Neural Tangent Kernel which is unable to explain predictions of finite width networks. Our convex geometric characterization also provides intuitive explanations of hidden neurons as auto-encoders. In higher dimensions, we show that the training problem can be cast as a finite dimensional convex problem with infinitely many constraints. Then, we apply certain convex relaxations and introduce a cutting-plane algorithm to globally optimize the network. We further analyze the exactness of the relaxations to provide conditions for the convergence to a global optimum. Our analysis also shows that optimal network parameters can be also characterized as interpretable closed-form formulas in some practically relevant special cases.

Key facts. Cited by 49 papers in the map (data through 13 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 12; 2024: 6; 2025: 5; 2026: 7.

49 papers.

Sort:

By area:

  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  2. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  3. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  4. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  5. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  6. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  7. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  8. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  9. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 18
  10. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  11. 2021Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutionscited by 14
  12. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
37 more
  1. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  2. 2022Convexifying Transformers: Improving optimization and understanding of transformer networkscited by 12
  3. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  4. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  5. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  6. 2021Parallel Deep Neural Networks Have Zero Duality Gapcited by 9
  7. 2023Globally Optimal Training of Neural Networks with Threshold Activation Functionscited by 9
  8. 2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
  9. 2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
  10. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  11. 2023Efficient displacement convex optimization with particle gradient descentcited by 5
  12. 2023From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexitycited by 4
  13. 2022Information Processing Equalities and the Information-Risk Bridgecited by 3
  14. 2022Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recoverycited by 3
  15. 2025Computational Advantages of Multi-Grade Deep Learning: Convergence Analysis and Performance Insightscited by 3
  16. 2022PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networkscited by 2
  17. 2023Polynomial-Time Solutions for ReLU Network Training: A Complexity Classification via Max-Cut and Zonotopescited by 2
  18. 2021Supervising the Decoder of Variational Autoencoders to Improve Scientific Utilitycited by 1
  19. 2023Fixing the NTK: From Neural Network Linearizations to Exact Convex Programscited by 1
  20. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  21. 2024Towards a Sampling Theory for Implicit Neural Representationscited by 1
  22. 2026On damage of interpolation to adversarial robustness in regressioncited by 1
  23. 2026Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splittingcited by 1
  24. 2026Online Realizable Regression and Applications for ReLU Networkscited by 1
  25. 2023Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholdingcited by 0
  26. 2024Dual feature-based and example-based explanation methodscited by 0
  27. 2024Probing Implicit Bias in Semi-gradient Q-learning: Visualizing the Effective Loss Landscapes via the Fokker--Planck Equationcited by 0
  28. 2024Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descentcited by 0
  29. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0
  30. 2025Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networkscited by 0
  31. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 0
  32. 2025A Recovery Guarantee for Sparse Neural Networkscited by 0
  33. 2025QML-HCS: A Hypercausal Quantum Machine Learning Framework for Non-Stationary Environmentscited by 0
  34. 2026Why ReLU? A Bit-Model Dichotomy for Deep Network Trainingcited by 0
  35. 2026Convex Optimization of Spiking Neural Networks with Global and Approximation Guaranteescited by 0
  36. 2026Exact Convex Reformulations of Linear Neural Networks via Completely Positive Liftingcited by 0
  37. 2026Convex Low-resource Accent-Robust Language Detection in Speech Recognitioncited by 0