mlsanityData to 14 Oct 2026

Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate

arXiv 2605.25134 ↗First posted 24 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is ℓp regularization. However, it may encounter optimization instability due to the unbounded gradients when 0<p<1. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to ℓp-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the ℓ1-regularization approach while preserving test accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 98 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

48 of 98 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  8. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  9. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  10. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  11. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  12. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
36 more
  1. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  2. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  3. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  4. 2019How do infinite width bounded norm networks look in function space?cited by 106
  5. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  6. 2014Matrix Completion and Low-Rank SVD via Fast Alternating Least Squarescited by 104
  7. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  8. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  9. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  10. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  11. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  12. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  13. 2016A survey of sparse representation: algorithms and applicationscited by 58
  14. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  15. 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
  16. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  17. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  18. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  19. 2020Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterizationcited by 26
  20. 2019High-Dimensional Linear Regression via Implicit Regularizationcited by 25
  21. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 24
  22. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  23. 2018A Survey on Nonconvex Regularization Based Sparse and Low-Rank Recovery in Signal Processing, Statistics, and Machine Learningcited by 20
  24. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  25. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  26. 2022spred: Solving L1 Penalty with SGDcited by 17
  27. 2021Smooth Bilevel Programming for Sparse Regularizationcited by 14
  28. 2024Mask in the Mirror: Implicit Sparsificationcited by 14
  29. 2020Understanding Implicit Regularization in Over-Parameterized Single Index Modelcited by 12
  30. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 12
  31. 2022Smooth over-parameterized solvers for non-smooth structured optimizationcited by 11
  32. 2019A Survey on Compressive Sensing: Classical Results and Recent Advancementscited by 6
  33. 2020Hadamard Wirtinger Flow for Sparse Phase Retrievalcited by 6
  34. 2025Sign-In to the Lottery: Reparameterizing Sparse Training From Scratchcited by 6
  35. 2020Improving Network Slimming with Nonconvex Regularizationcited by 5
  36. 2022Blessing of Nonconvexity in Deep Linear Models: Depth Flattens the Optimization Landscape Around the True Solutioncited by 3

Built on by

Not cited yet.