mlsanityData to 14 Oct 2026

Soft Threshold Weight Reparameterization for Learnable Sparsity

arXiv 2002.03231 ↗First posted 8 Feb 2020cs.LG cs.CV stat.MLcited by 151

Not on the map. Why?

Abstract (arXiv)

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FLOPs). This work proposes Soft Threshold Reparameterization (STR), a novel use of the soft-threshold operator on DNN weights. STR smoothly induces sparsity while learning pruning thresholds thereby obtaining a non-uniform sparsity budget. Our method achieves state-of-the-art accuracy for unstructured sparsity in CNNs (ResNet50 and MobileNetV1 on ImageNet-1K), and, additionally, learns non-uniform budgets that empirically reduce the FLOPs by up to 50%. Notably, STR boosts the accuracy over existing results by up to 10% in the ultra sparse (99%) regime and can also be used to induce low-rank (structured sparsity) in RNNs. In short, STR is a simple mechanism which learns effective sparsity budgets that contrast with popular heuristics. Code, pretrained models and sparsity budgets are at https://github.com/RAIVNLab/STR.

Key facts. Cited by 151 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 29; 2022: 34; 2023: 34; 2024: 29; 2025: 18; 2026: 7.

151 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2022Matryoshka Representation Learningcited by 282
  3. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  4. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  5. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  6. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  7. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 85
  8. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  9. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  10. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  11. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  12. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
38 more
  1. 2022Advancing Model Pruning via Bi-level Optimizationcited by 56
  2. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  3. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  4. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  5. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  6. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  7. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43
  8. 2021Learnable Embedding Sizes for Recommender Systemscited by 42
  9. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  10. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 36
  11. 2022Learning Best Combination for Efficient N:M Sparsitycited by 32
  12. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  13. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  14. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 29
  15. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  16. 2021Selfish Sparse RNN Trainingcited by 26
  17. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 26
  18. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  19. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  20. 2021Post-training deep neural network pruning via layer-wise calibrationcited by 24
  21. 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
  22. 2024Pruning for Robust Concept Erasing in Diffusion Modelscited by 22
  23. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  24. 2022Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptionscited by 20
  25. 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
  26. 2023Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inferencecited by 20
  27. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  28. 2023Embedding in Recommender Systems: A Surveycited by 20
  29. 2024ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Modelscited by 19
  30. 2022Towards Sparsification of Graph Neural Networkscited by 18
  31. 2023Beyond Classical Attention: Quantum Attention for Scalable Computationcited by 18
  32. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 18
  33. 2021An Information Theory-inspired Strategy for Automatic Network Pruningcited by 17
  34. 2022spred: Solving L1 Penalty with SGDcited by 17
  35. 2021Truly Sparse Neural Networks at Scalecited by 16
  36. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
  37. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  38. 2024Masks, Signs, And Learning Rate Rewindingcited by 15

50 most cited of 151 All 151 (JSON)