# spred: Solving L1 Penalty with SGD

arXiv: https://arxiv.org/abs/2210.01212 · HTML: https://mlsanity.com/paper/2210.01212/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [Soft Threshold Weight Reparameterization for Learnable Sparsity](https://mlsanity.com/paper/2002.03231/) (2020)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Gradient Descent Can Take Exponential Time to Escape Saddle Points](https://mlsanity.com/paper/1705.10412/) (2017)
- [Discovering Neural Wirings](https://mlsanity.com/paper/1906.00586/) (2019)
- [Smooth Bilevel Programming for Sparse Regularization](https://mlsanity.com/paper/2106.01429/) (2021)
- [Smooth over-parameterized solvers for non-smooth structured optimization](https://mlsanity.com/paper/2205.01385/) (2022)

## Built on by (most cited first, 17 of 17)

- [Weight decay induces low-rank attention layers](https://mlsanity.com/paper/2410.23819/) (2024)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries](https://mlsanity.com/paper/2502.02496/) (2025)
- [How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective](https://mlsanity.com/paper/2306.05857/) (2023)
- [Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent](https://mlsanity.com/paper/2303.13093/) (2023)
- [Remove Symmetries to Control Model Expressivity and Improve Optimization](https://mlsanity.com/paper/2408.15495/) (2024)
- [Never Saddle for Reparameterized Steepest Descent as Mirror Flow](https://mlsanity.com/paper/2603.02064/) (2026)
- [Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization](https://mlsanity.com/paper/2509.23898/) (2025)
- [Weight-Sharing Regularization](https://mlsanity.com/paper/2311.03096/) (2023)
- [CoNNect: Connectivity-Based Regularization for Structural Pruning](https://mlsanity.com/paper/2502.00744/) (2025)
- [Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networks](https://mlsanity.com/paper/2505.21791/) (2025)
- [Addressing Instrument-Outcome Confounding in Mendelian Randomization through Representation Learning](https://mlsanity.com/paper/2602.19782/) (2026)
- [Task Switching Without Forgetting via Proximal Decoupling](https://mlsanity.com/paper/2604.18857/) (2026)
- [HORST: Composing Optimizer Geometries for Sparse Transformer Training](https://mlsanity.com/paper/2605.21104/) (2026)
- [Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate](https://mlsanity.com/paper/2605.25134/) (2026)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)
- [Mind the Drift: Diagonal Linear Networks Under Large Learning Rates](https://mlsanity.com/paper/2610.06120/) (2026)

## Abstract (arXiv)

We propose to minimize a generic differentiable objective with L1 constraint using a simple reparametrization and straightforward stochastic gradient descent. Our proposal is the direct generalization of previous ideas that the L1 penalty may be equivalent to a differentiable reparametrization with weight decay. We prove that the proposed method, textitspred, is an exact differentiable solver of L1 and that the reparametrization trick is completely ``benign" for a generic nonconvex function. Practically, we demonstrate the usefulness of the method in (1) training sparse neural networks to perform gene selection tasks, which involves finding relevant features in a very high dimensional space, and (2) neural network compression task, to which previous attempts at applying the L1-penalty have been unsuccessful. Conceptually, our result bridges the gap between the sparsity in deep learning and conventional statistical learning.