mlsanityData to 7 Oct 2026

Accelerating DNN Training with Structured Data Gradient Pruning

arXiv 2202.00774 ↗First posted 1 Feb 2022cs.LG cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Weight pruning is a technique to make Deep Neural Network (DNN) inference more computationally efficient by reducing the number of model parameters over the course of training. However, most weight pruning techniques generally does not speed up DNN training and can even require more iterations to reach model convergence. In this work, we propose a novel Structured Data Gradient Pruning (SDGP) method that can speed up training without impacting model convergence. This approach enforces a specific sparsity structure, where only N out of every M elements in a matrix can be nonzero, making it amenable to hardware acceleration. Modern accelerators such as the Nvidia A100 GPU support this type of structured sparsity for 2 nonzeros per 4 elements in a reduction. Assuming hardware support for 2:4 sparsity, our approach can achieve a 15-25% reduction in total training time without significant impact to performance. Source code and pre-trained models are available at urlhttps://github.com/BradMcDanel/sdgp.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

24 of 36 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2020Designing Network Design Spacescited by 954
  9. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  11. 2018Rethinking the Value of Network Pruningcited by 595
  12. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
12 more
  1. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  2. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  3. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  4. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  5. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  6. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  7. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  8. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  9. 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
  10. 2020Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Trainingcited by 18
  11. 2020TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inferencecited by 16
  12. 2018Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimizationcited by 15

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 3.

8 papers.

  1. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  2. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 18
  3. 2022Minimum Variance Unbiased N:M Sparsity for the Neural Gradientscited by 9
  4. 2023Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Designcited by 5
  5. 2023PruMUX: Augmenting Data Multiplexing with Model Compressioncited by 3
  6. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  7. 2025Study of Training Dynamics for Memory-Constrained Fine-Tuningcited by 1
  8. 2025Memory Constrained Dynamic Subnetwork Update for Transfer Learningcited by 0