mlsanityData to 14 Oct 2026

Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks

arXiv 2110.09548 ↗First posted 18 Oct 2021cs.LG cs.AI stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

36 of 63 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2016Wide Residual Networkscited by 3,070
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2021Convex optimizationcited by 2,515
  6. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  7. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  8. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  9. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  10. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  12. 2015Norm-Based Capacity Control in Neural Networkscited by 299
24 more
  1. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  2. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  3. 2017A Rewriting System for Convex Optimization Problemscited by 179
  4. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  5. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  6. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
  7. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  8. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  9. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  10. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
  11. 2017Failures of Gradient-Based Deep Learningcited by 80
  12. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  13. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  14. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  15. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  16. 2020Tight Hardness Results for Training Depth-2 ReLU Networkscited by 35
  17. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 34
  18. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  19. 2016Efficient approaches for escaping higher order saddle points in non-convex optimizationcited by 23
  20. 2021The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionalitycited by 22
  21. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  22. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  23. 2021Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutionscited by 14
  24. 2021Parallel Deep Neural Networks Have Zero Duality Gapcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 3; 2025: 2; 2026: 2.

14 papers.

Sort:

By area:

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  2. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  3. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  4. 2023Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrizationcited by 11
  5. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  6. 2023Globally Optimal Training of Neural Networks with Threshold Activation Functionscited by 9
  7. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 8
  8. 2024Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimizationcited by 7
  9. 2024Geometric Algebra Planes: Convex Implicit Neural Volumescited by 5
  10. 2023Fixing the NTK: From Neural Network Linearizations to Exact Convex Programscited by 1
  11. 2026Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspectivecited by 1
  12. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
2 more
  1. 2025A Recovery Guarantee for Sparse Neural Networkscited by 0
  2. 2026Convex Optimization of Spiking Neural Networks with Global and Approximation Guaranteescited by 0