mlsanityData to 14 Oct 2026

Implicit Regularization for Group Sparsity

arXiv 2301.12540 ↗First posted 29 Jan 2023stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We study the implicit regularization of gradient descent towards structured sparsity via a novel neural reparameterization, which we call a diagonally grouped linear neural network. We show the following intriguing property of our reparameterization: gradient descent over the squared regression loss, without any explicit regularization, biases towards solutions with a group sparsity structure. In contrast to many existing works in understanding implicit regularization, we prove that our training trajectory cannot be simulated by mirror descent. We analyze the gradient dynamics of the corresponding regression problem in the general noise setting and obtain minimax-optimal error rates. Compared to existing bounds for implicit sparse regularization using diagonal linear networks, our analysis with the new reparameterization shows improved sample complexity. In the degenerate case of size-one groups, our approach gives rise to a new algorithm for sparse linear regression. Finally, we demonstrate the efficacy of our approach with several numerical experiments.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 29 Jan 2023.

Builds on

31 of 45 references.

References · 31

  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  3. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  4. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  6. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  11. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  12. 2018Convergence of Gradient Descent on Separable Datacited by 110
19 more
  1. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  2. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
  3. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  4. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  5. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  6. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  7. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  8. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  9. 2020Implicit Rank-Minimizing Autoencodercited by 35
  10. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 34
  11. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  12. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  13. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 24
  14. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  15. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  16. 2020Beyond Lazy Training for Over-parameterized Tensor Decompositioncited by 15
  17. 2020Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Netscited by 5
  18. 2021yaglm: a Python package for fitting and tuning generalized linear models that supports structured, adaptive and non-convex penaltiescited by 2
  19. 2017Convolution with Logarithmic Filter Groups for Efficient Shallow CNNcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Mask in the Mirror: Implicit Sparsificationcited by 14
  2. 2025Sign-In to the Lottery: Reparameterizing Sparse Training From Scratchcited by 6
  3. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
  4. 2025Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalizationcited by 3
  5. 2024Precise asymptotics of reweighted least-squares algorithms for linear diagonal networkscited by 2
  6. 2026Incremental Learning in Mirror Flowscited by 2
  7. 2024Tuning-Free Online Robust Principal Component Analysis through Implicit Regularizationcited by 0
  8. 2025Tuning-Free Structured Sparse Recovery of Multiple Measurement Vectors using Implicit Regularizationcited by 0