mlsanityData to 14 Oct 2026

Always-Sparse Training by Growing Connections with Guided Stochastic Exploration

arXiv 2401.06898 ↗First posted 12 Jan 2024cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

The excessive computational requirements of modern artificial neural networks (ANNs) are posing limitations on the machines that can run them. Sparsification of ANNs is often motivated by time, memory and energy savings only during model inference, yielding no benefits during training. A growing body of work is now focusing on providing the benefits of model sparsification also during training. While these methods greatly improve the training efficiency, the training algorithms yielding the most accurate models still materialize the dense weights, or compute dense gradients during training. We propose an efficient, always-sparse training algorithm with excellent scaling to larger and sparser models, supported by its linear time complexity with respect to the model width during training and inference. Moreover, our guided stochastic exploration algorithm improves over the accuracy of previous sparse training methods. We evaluate our method on CIFAR-10/100 and ImageNet using ResNet, VGG, and ViT models, and compare it against a range of sparsification methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 55 of its 84 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

55 of 84 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
38 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  2. 2018Rethinking the Value of Network Pruningcited by 599
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  4. 2020What is the State of Neural Network Pruning?cited by 583
  5. 2019Importance Estimation for Neural Network Pruningcited by 541
  6. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  8. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  9. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  11. 2019The State of Sparsity in Deep Neural Networkscited by 476
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  13. 2015Deep Learning with Limited Numerical Precisioncited by 376
  14. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  15. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  16. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  17. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  18. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  19. 2015Tensorizing Neural Networkscited by 250
  20. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  21. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  22. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 221
  23. 2018Efficient Neural Audio Synthesiscited by 219
  24. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  25. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  26. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  27. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  28. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  29. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  30. 2019Deep Learning vs. Traditional Computer Visioncited by 94
  31. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  32. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  33. 2020The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Researchcited by 67
  34. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  35. 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
  36. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  37. 2017WRPN: Wide Reduced-Precision Networkscited by 51
  38. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Gradual Capacity Growth for Sparse Network Discoverycited by 0