mlsanityData to 7 Oct 2026

Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks

arXiv 2002.10553 ↗First posted 24 Feb 2020cs.LG cs.CC stat.MLcited by 96

Not on the map. Why?

Abstract (arXiv)

We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons. Our theory utilizes semi-infinite duality and minimum norm regularization. We show that ReLU networks trained with standard weight decay are equivalent to block ℓ1 penalized convex models. Moreover, we show that certain standard convolutional linear networks are equivalent semi-definite programs which can be simplified to ℓ1 regularized linear models in a polynomial sized discrete Fourier feature space.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 17; 2023: 17; 2024: 22; 2025: 13; 2026: 14.

96 papers.

Sort:

By area:

  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  2. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  3. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  4. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  5. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  6. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  7. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  8. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  9. 2023Training Neural Networks is NP-Hard in Fixed Dimensioncited by 23
  10. 2021The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionalitycited by 22
  11. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  12. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
38 more
  1. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  2. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  3. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  4. 2022Efficient Global Optimization of Two-Layer ReLU Networks: Quadratic-Time Algorithms and Adversarial Trainingcited by 15
  5. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  6. 2021Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutionscited by 14
  7. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  8. 2022Convexifying Transformers: Improving optimization and understanding of transformer networkscited by 12
  9. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  10. 2024Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Timecited by 11
  11. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  12. 2021Parallel Deep Neural Networks Have Zero Duality Gapcited by 9
  13. 2022Optimal Neural Network Approximation of Wasserstein Gradient Direction via Convex Optimizationcited by 9
  14. 2023Globally Optimal Training of Neural Networks with Threshold Activation Functionscited by 9
  15. 2023Optimal Sets and Solution Paths of ReLU Networkscited by 9
  16. 2023Implicit Regularization for Group Sparsitycited by 8
  17. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  18. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  19. 2022CoNSoLe: Convex Neural Symbolic Learningcited by 7
  20. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  21. 2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
  22. 2024Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimizationcited by 7
  23. 2021Training Quantized Neural Networks to Global Optimality via Semidefinite Programmingcited by 6
  24. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  25. 2023On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrizationcited by 5
  26. 2024Geometric Algebra Planes: Convex Implicit Neural Volumescited by 5
  27. 2021Practical Convex Formulation of Robust One-hidden-layer Neural Network Trainingcited by 4
  28. 2023From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexitycited by 4
  29. 2024CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networkscited by 4
  30. 2025SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learningcited by 4
  31. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  32. 2022Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recoverycited by 3
  33. 2022A Combinatorial Perspective on the Optimization of Shallow ReLU Networkscited by 3
  34. 2022On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrizationcited by 3
  35. 2023FairDP: Certified Fairness with Differential Privacycited by 3
  36. 2025Computational Advantages of Multi-Grade Deep Learning: Convergence Analysis and Performance Insightscited by 3
  37. 2021Nonasymptotic theory for two-layer neural networks: Beyond the bias-variance trade-offcited by 2
  38. 2021Integral representations of shallow neural network with Rectified Power Unit activation functioncited by 2

50 most cited of 96 All 96 (JSON)