mlsanityData to 7 Oct 2026

SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks

arXiv 2302.04852 ↗First posted 9 Feb 2023cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

We provide a new efficient version of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and common layer types (e.g., convolutional or linear). We provide a fast vectorized implementation on commodity CPUs, and show that it can yield speedups in end-to-end runtime experiments, both in transfer learning using already-sparsified networks, and in training sparse networks from scratch. Thus, our results provide the first support for sparse training on commodity hardware.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 44 of its 69 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

44 of 69 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
32 more
  1. 2018Do Better ImageNet Models Transfer Better?cited by 560
  2. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  3. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  4. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  5. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  6. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  7. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  8. 2021Accelerating Sparse Deep Neural Networkscited by 213
  9. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  10. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  11. 2021Block Pruning For Faster Transformerscited by 176
  12. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  13. 2020Sparse GPU Kernels for Deep Learningcited by 138
  14. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  15. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  16. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  17. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  18. 2019Fast Sparse ConvNetscited by 86
  19. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  20. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  21. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  22. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  23. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  24. 2020Sparse Weight Activation Trainingcited by 45
  25. 2016Faster CNNs with Direct Sparse Convolutions and Guided Pruningcited by 39
  26. 2016A topological insight into restricted Boltzmann machinescited by 32
  27. 2020SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inferencecited by 30
  28. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  29. 2020Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Trainingcited by 18
  30. 2018Full deep neural network training on a pruned weight budgetcited by 11
  31. 2020Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network trainingcited by 11
  32. 2019Memorized Sparse Backpropagationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 3; 2026: 1.

12 papers.

  1. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  2. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  3. 2024SAFT: Towards Out-of-Distribution Generalization in Fine-Tuningcited by 12
  4. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  5. 2024TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generationcited by 5
  6. 2023Beyond Multilayer Perceptrons: Investigating Complex Topologies in Neural Networkscited by 3
  7. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  8. 2024Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagationcited by 3
  9. 2026Sparse Training of Neural Networks based on Multilevel Mirror Descentcited by 3
  10. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruningcited by 1
  11. 2025Finite-Agent Stochastic Differential Games on Large Graphs: II. Graph-Based Architecturescited by 1
  12. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0