mlsanityData to 14 Oct 2026

AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks

arXiv 2304.06941 ↗First posted 14 Apr 2023cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Sparse training is emerging as a promising avenue for reducing the computational cost of training neural networks. Several recent studies have proposed pruning methods using learnable thresholds to efficiently explore the non-uniform distribution of sparsity inherent within the models. In this paper, we propose Gradient Annealing (GA), where gradients of masked weights are scaled down in a non-linear manner. GA provides an elegant trade-off between sparsity and accuracy without the need for additional sparsity-inducing regularization. We integrated GA with the latest learnable pruning methods to create an automated sparse training algorithm called AutoSparse, which achieves better accuracy and/or training/inference FLOPS reduction than existing learnable pruning methods for sparse ResNet50 and MobileNetV1 on ImageNet-1K: AutoSparse achieves (2x, 7x) reduction in (training,inference) FLOPS for ResNet50 on ImageNet at 80% sparsity. Finally, AutoSparse outperforms sparse-to-sparse SotA method MEST (uniform sparsity) for 80% sparse ResNet50 with similar accuracy, where MEST uses 12% more training FLOPS and 50% more inference FLOPS.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

41 of 50 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  12. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
29 more
  1. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  2. 2019The State of Sparsity in Deep Neural Networkscited by 476
  3. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  4. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  6. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  7. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  8. 2021Accelerating Sparse Deep Neural Networkscited by 223
  9. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  10. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  11. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  12. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  13. 2020Dynamic Model Pruning with Feedbackcited by 125
  14. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  15. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  16. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
  17. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  18. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 68
  19. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  20. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
  21. 2019Discovering Neural Wiringscited by 56
  22. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  23. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  24. 2020Sparse Weight Activation Trainingcited by 45
  25. 2021GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarizationcited by 29
  26. 2019Mixed Precision Training With 8-bit Floating Pointcited by 23
  27. 2020Single Shot Structured Pruning Before Trainingcited by 19
  28. 2020Learned Threshold Pruningcited by 17
  29. 2022Automatic Sparse Connectivity Learning for Neural Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.

2 papers.

  1. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 17
  2. 2026PQuantML: A Tool for End-to-End Hardware-aware Model Compressioncited by 0