mlsanityData to 7 Oct 2026

PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks

arXiv 2210.03069 ↗First posted 6 Oct 2022cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Weight decay is one of the most widely used forms of regularization in deep learning, and has been shown to improve generalization and robustness. The optimization objective driving weight decay is a sum of losses plus a term proportional to the sum of squared weights. This paper argues that stochastic gradient descent (SGD) may be an inefficient algorithm for this objective. For neural networks with ReLU activations, solutions to the weight decay objective are equivalent to those of a different objective in which the regularization term is instead a sum of products of ℓ2 (not squared) norms of the input and output weights associated with each ReLU neuron. This alternative (and effectively equivalent) regularization suggests a novel proximal gradient algorithm for network training. Theory and experiments support the new training approach, showing that it can converge much faster to the sparse solutions it shares with standard weight decay training.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

39 of 63 references.

References · 39

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  8. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  9. 2020Bag of Tricks for Adversarial Trainingcited by 199
  10. 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
  11. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  12. 2018Stochastic subgradient method converges on tame functionscited by 120
27 more
  1. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  2. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  3. 2019How do infinite width bounded norm networks look in function space?cited by 105
  4. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  5. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  6. 2020In Search of Robust Measures of Generalizationcited by 79
  7. 2018Sparse DNNs with Improved Adversarial Robustnesscited by 71
  8. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  9. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  10. 2016Loss-aware Binarization of Deep Networkscited by 56
  11. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 56
  12. 2020A law of robustness for two-layers neural networkscited by 54
  13. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  14. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  15. 2020Improving Generalization by Controlling Label-Noise Information in Neural Network Weightscited by 35
  16. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  17. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  18. 2020Orthant Based Proximal Stochastic Gradient Method for ℓ1-Regularized Optimizationcited by 19
  19. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  20. 2019Equi-normalization of Neural Networkscited by 16
  21. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  22. 2020Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizationscited by 9
  23. 2019Exploring Structural Sparsity of Deep Networks via Inverse Scale Spacescited by 7
  24. 2021Training Structured Neural Networks Through Manifold Identification and Variance Reductioncited by 7
  25. 2020A General Family of Stochastic Proximal Gradient Methods for Deep Learningcited by 6
  26. 2018Adversarial Training Versus Weight Decaycited by 3
  27. 2020Efficient Proximal Mapping of the 1-path-norm of Shallow Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecturecited by 0
  2. 2025Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networkscited by 0