mlsanityData to 7 Oct 2026

Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training?

arXiv 2212.01076 ↗First posted 2 Dec 2022cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Turning the weights to zero when training a neural network helps in reducing the computational complexity at inference. To progressively increase the sparsity ratio in the network without causing sharp weight discontinuities during training, our work combines soft-thresholding and straight-through gradient estimation to update the raw, i.e. non-thresholded, version of zeroed weights. Our method, named ST-3 for straight-through/soft-thresholding/sparse-training, obtains SoA results, both in terms of accuracy/sparsity and accuracy/FLOPS trade-offs, when progressively increasing the sparsity ratio in a single training cycle. In particular, despite its simplicity, ST-3 favorably compares to the most recent methods, adopting differentiable formulations or bio-inspired neuroregeneration principles. This suggests that the key ingredients for effective sparsification primarily lie in the ability to give the weights the freedom to evolve smoothly across the zero state while progressively increasing the sparsity ratio. Source code and weights available at https://github.com/vanderschuea/stthree

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

37 of 48 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2016Wide Residual Networkscited by 3,057
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  12. 2018Rethinking the Value of Network Pruningcited by 595
25 more
  1. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  2. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  3. 2019The State of Sparsity in Deep Neural Networkscited by 471
  4. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  5. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  6. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  7. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  8. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  9. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  10. 2018Efficient Neural Audio Synthesiscited by 218
  11. 2021Accelerating Sparse Deep Neural Networkscited by 213
  12. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  13. 2016Ternary Weight Networkscited by 195
  14. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  15. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  16. 2020The Early Phase of Neural Network Trainingcited by 114
  17. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  18. 2020Neural Pruning via Growing Regularizationcited by 97
  19. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  20. 2020EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruningcited by 80
  21. 2019Discovering Neural Wiringscited by 56
  22. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  23. 2016Faster CNNs with Direct Sparse Convolutions and Guided Pruningcited by 39
  24. 2020The Two Regimes of Deep Network Trainingcited by 30
  25. 2017Ternary Neural Networks with Fine-Grained Quantizationcited by 15

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 3.

9 papers.

  1. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  2. 2023Effective Neural Network L0 Regularization With BinMaskcited by 2
  3. 2024Edge Detectors Can Make Deep Convolutional Neural Networks More Robustcited by 2
  4. 2024AdapMTL: Adaptive Pruning Framework for Multitask Learning Modelcited by 2
  5. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 2
  6. 2023Feather: An Elegant Solution to Effective DNN Sparsificationcited by 1
  7. 2025CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Modelscited by 1
  8. 2024SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimizationcited by 0
  9. 2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learningcited by 0