mlsanityData to 14 Oct 2026

Learning Sparse Neural Networks through L0 Regularization

arXiv 1712.01312 ↗First posted 4 Dec 2017stat.ML cs.LGcited by 583

Not on the map. Why?

Abstract (arXiv)

We propose a practical method for L0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of L0 regularization. However, since the L0 norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected L0 norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the hard concrete distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

Key facts. Cited by 583 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 111; 2022: 108; 2023: 103; 2024: 81; 2025: 96; 2026: 84.

583 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  5. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  6. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  7. 2021Block Pruning For Faster Transformerscited by 177
  8. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  9. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  10. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  11. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  12. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
38 more
  1. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  2. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  3. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  4. 2021Survey of Low-Resource Machine Translationcited by 92
  5. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  6. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  7. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  8. 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
  9. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  10. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  11. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  12. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  13. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  14. 2021A Survey on Graph Structure Learning: Progress and Opportunitiescited by 71
  15. 2022Deep Differentiable Logic Gate Networkscited by 67
  16. 2021Low-Complexity Probing via Finding Subnetworkscited by 64
  17. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  18. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 62
  19. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  20. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  21. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  22. 2022Transferability in Deep Learning: A Surveycited by 57
  23. 2022Advancing Model Pruning via Bi-level Optimizationcited by 56
  24. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  25. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  26. 2022Online Continual Learning for Embedded Devicescited by 49
  27. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  28. 2023Intriguing Properties of Quantization at Scalecited by 49
  29. 2021Differentiable Subset Pruning of Transformer Headscited by 47
  30. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
  31. 2023Pruning vs Quantization: Which is Better?cited by 46
  32. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  33. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  34. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  35. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  36. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 44
  37. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  38. 2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 44

50 most cited of 583 All 583 (JSON)