mlsanityData to 7 Oct 2026

Dynamic Sparse Training with Structured Sparsity

arXiv 2305.02299 ↗First posted 3 May 2023cs.LG cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting models are highly sparse and theoretically less computationally expensive, achieving speedups with unstructured sparsity on real-world hardware is challenging. In this work, we propose a sparse-to-sparse DST method, Structured RigL (SRigL), to learn a variant of fine-grained structured N:M sparsity by imposing a constant fan-in constraint. Using our empirical analysis of existing DST methods at high sparsity, we additionally employ a neuron ablation method which enables SRigL to achieve state-of-the-art sparse-to-sparse structured DST performance on a variety of Neural Network (NN) architectures. Using a 90% sparse linear layer, we demonstrate a real-world acceleration of 3.4x/2.5x on CPU for online inference and 1.7x/13.0x on GPU for inference with a batch size of 256 when compared to equivalent dense/unstructured (CSR) sparse layers, respectively.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

51 of 70 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016Wide Residual Networkscited by 3,057
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
38 more
  1. 2019Searching for MobileNetV3cited by 1,984
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2017Random Erasing Data Augmentationcited by 1,235
  4. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  6. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  7. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  8. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  10. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  11. 2021Accelerating Sparse Deep Neural Networkscited by 213
  12. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  13. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  14. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  15. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
  16. 2020Sparse GPU Kernels for Deep Learningcited by 138
  17. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  18. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  19. 2019Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networkscited by 118
  20. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  21. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
  22. 2019Fast Sparse ConvNetscited by 86
  23. 2021Sparse is Enough in Scaling Transformerscited by 86
  24. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  25. 2019Network Pruning via Transformable Architecture Searchcited by 80
  26. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  27. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  28. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  29. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  30. 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
  31. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  32. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  33. 2021Locally Free Weight Sharing for Network Width Searchcited by 27
  34. 2022Merlin HugeCTR: GPU-accelerated Recommender System Training and Inferencecited by 20
  35. 2021Towards Structured Dynamic Sparse Pre-Training of BERTcited by 12
  36. 2023SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networkscited by 12
  37. 2020Accelerating Deep Learning Inference via Freezingcited by 10
  38. 2020Collegial Ensemblescited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 11.

36 papers.

Sort:

By area:

  1. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  2. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  3. 2024Beware of Calibration Data for Pruning Large Language Modelscited by 14
  4. 2024Mask in the Mirror: Implicit Sparsificationcited by 12
  5. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  6. 2025TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networkscited by 9
  7. 2025Sign-In to the Lottery: Reparameterizing Sparse Training From Scratchcited by 6
  8. 2024Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neuronscited by 5
  9. 2024SCATTER: Algorithm-Circuit Co-Sparse Photonic Accelerator with Thermal-Tolerant, Power-Efficient In-situ Light Redistributioncited by 5
  10. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  11. 2024Learning Fine-grained Parameter Sharing via Sparse Tensor Decompositioncited by 3
  12. 2025Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connectedcited by 3
24 more
  1. 2025Dynamic Sparse Training of Diagonally Sparse Networkscited by 3
  2. 2025Symmetric Pruning of Large Language Modelscited by 2
  3. 2025SparseLUT: Sparse Connectivity Optimization for Lookup Table-based Deep Neural Networkscited by 2
  4. 2025LoKI: Low-damage Knowledge Implanting of Large Language Modelscited by 2
  5. 2026Bayesian Lottery Ticket Hypothesiscited by 2
  6. 2025Advancing Weight and Channel Sparsification with Enhanced Saliencycited by 1
  7. 2025SD2: Self-Distilled Sparse Drafterscited by 1
  8. 2025Accelerating Learned Image Compression Through Modeling Neural Training Dynamicscited by 1
  9. 2025ESLM: Risk-Averse Selective Language Modeling for Efficient Pretrainingcited by 1
  10. 2025Theoretical Compression Bounds for Wide Multilayer Perceptronscited by 1
  11. 2024Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruningcited by 0
  12. 2025Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNscited by 0
  13. 2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learningcited by 0
  14. 2025SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networkscited by 0
  15. 2026Enhancing LUT-based Deep Neural Networks Inference through Architecture and Connectivity Optimizationcited by 0
  16. 2026SAL: Selective Adaptive Learning for Backpropagation-Free Training with Sparsificationcited by 0
  17. 2026An Adaptive Differentially Private Federated Learning Frameworkcited by 0
  18. 2026Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibriacited by 0
  19. 2026On the Stability of Growth in Structural Plasticity: Forward-Active yet Backward-Starvedcited by 0
  20. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  21. 2026SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Trainingcited by 0
  22. 2026Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scalingcited by 0
  23. 2026HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spacescited by 0
  24. 2026Modularity-Free Conflict-Averse Training for Generalized PINNscited by 0