# Rethinking Weight Decay For Efficient Neural Network Pruning

arXiv: https://arxiv.org/abs/2011.10520 · HTML: https://mlsanity.com/paper/2011.10520/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 10 of 10)

- [Towards Sparsification of Graph Neural Networks](https://mlsanity.com/paper/2209.04766/) (2022)
- [Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers](https://mlsanity.com/paper/2402.04744/) (2024)
- [Can Interpretation Predict Behavior on Unseen Data?](https://mlsanity.com/paper/2507.06445/) (2025)
- [A Strong and Simple Deep Learning Baseline for BCI MI Decoding](https://mlsanity.com/paper/2309.07159/) (2023)
- [FLoCoRA: Federated learning compression with low-rank adaptation](https://mlsanity.com/paper/2406.14082/) (2024)
- [Federated learning compression designed for lightweight communications](https://mlsanity.com/paper/2310.14693/) (2023)
- [Characterising the Inductive Biases of Neural Networks on Boolean Data](https://mlsanity.com/paper/2505.24060/) (2025)
- [Pruning Graph Convolutional Networks to select meaningful graph frequencies for fMRI decoding](https://mlsanity.com/paper/2203.04455/) (2022)
- [Effective Multi-Stage Training Model For Edge Computing Devices In Intrusion Detection](https://mlsanity.com/paper/2401.17546/) (2024)
- [Adaptive Training of INRs via Pruning and Densification](https://mlsanity.com/paper/2510.23943/) (2025)

## Abstract (arXiv)

Introduced in the late 1980s for generalization purposes, pruning has now become a staple for compressing deep neural networks. Despite many innovations in recent decades, pruning approaches still face core issues that hinder their performance or scalability. Drawing inspiration from early work in the field, and especially the use of weight decay to achieve sparsity, we introduce Selective Weight Decay (SWD), which carries out efficient, continuous pruning throughout training. Our approach, theoretically grounded on Lagrangian smoothing, is versatile and can be applied to multiple tasks, networks, and pruning structures. We show that SWD compares favorably to state-of-the-art approaches, in terms of performance-to-parameters ratio, on the CIFAR-10, Cora, and ImageNet ILSVRC2012 datasets.