mlsanityData to 7 Oct 2026

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

arXiv 2106.10404 ↗First posted 19 Jun 2021cs.LG cs.CVcited by 83

Not on the map. Why?

Abstract (arXiv)

Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training pruning, we quantitatively study the effect of pruning throughout training from the perspective of pruning plasticity (the ability of the pruned networks to recover the original performance). Pruning plasticity can help explain several other empirical observations about neural network pruning in literature. We further find that pruning plasticity can be substantially improved by injecting a brain-inspired mechanism called neuroregeneration, i.e., to regenerate the same number of connections as pruned. We design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (GraNet), that advances state of the art. Perhaps most impressively, its sparse-to-sparse version for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods with ResNet-50 on ImageNet without extending the training time. We release all codes in https://github.com/Shiweiliuiiiiiii/GraNet.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 19 Jun 2021.

Builds on

61 of 79 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  9. 2019Green AIcited by 620
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  11. 2018Rethinking the Value of Network Pruningcited by 595
  12. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
38 more
  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  2. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  3. 2019Importance Estimation for Neural Network Pruningcited by 531
  4. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  5. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  6. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  7. 2019The State of Sparsity in Deep Neural Networkscited by 471
  8. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  10. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  11. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  12. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  13. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  14. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  15. 2019Fixing the train-test resolution discrepancycited by 209
  16. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  17. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  18. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  19. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  20. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  21. 2020Sparse GPU Kernels for Deep Learningcited by 138
  22. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  23. 2020Dynamic Model Pruning with Feedbackcited by 125
  24. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  25. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  26. 2019Winning the Lottery with Continuous Sparsificationcited by 99
  27. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  28. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  29. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  30. 2016Training Sparse Neural Networkscited by 73
  31. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  32. 2020Progressive Skeletonization: Trimming more fat from a network at initializationcited by 64
  33. 2019Discovering Neural Wiringscited by 56
  34. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  35. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  36. 2019Stabilizing the Lottery Ticket Hypothesiscited by 36
  37. 2020Fixing the train-test resolution discrepancy: FixEfficientNetcited by 31
  38. 2020Finding trainable sparse networks through Neural Tangent Transfercited by 28

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 22; 2023: 33; 2024: 10; 2025: 12; 2026: 2.

83 papers.

Sort:

By area:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  3. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  4. 2023Deep Model Fusion: A Surveycited by 89
  5. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  6. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  7. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  8. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  9. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
  10. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  11. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  12. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
38 more
  1. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  2. 2022Learning Best Combination for Efficient N:M Sparsitycited by 32
  3. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  4. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  5. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  6. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  7. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  8. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 26
  9. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  10. 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
  11. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  12. 2026Flow caching for autoregressive video generationcited by 19
  13. 2022Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networkscited by 18
  14. 2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
  15. 2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
  16. 2023E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentationcited by 16
  17. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  18. 2023A Three-regime Model of Network Pruningcited by 15
  19. 2022You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Ticketscited by 14
  20. 2023Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detectioncited by 14
  21. 2021Plant 'n' Seek: Can You Find the Winning Ticket?cited by 13
  22. 2023A Unified Framework for Soft Threshold Pruningcited by 12
  23. 2023PDP: Parameter-free Differentiable Pruning is All You Needcited by 12
  24. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  25. 2024Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustnesscited by 11
  26. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  27. 2022Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithmcited by 9
  28. 2022Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training?cited by 9
  29. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  30. 2023Balanced Training for Sparse GANscited by 8
  31. 2025Efficient Shapley Value-based Non-Uniform Pruning of Large Language Modelscited by 8
  32. 2023Supervised Feature Selection with Neuron Evolution in Sparse Neural Networkscited by 7
  33. 2023Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiencycited by 7
  34. 2022Signing the Supermask: Keep, Hide, Invertcited by 6
  35. 2022Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Trainingcited by 6
  36. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  37. 2022OptG: Optimizing Gradient-driven Criteria in Network Sparsitycited by 5
  38. 2023Magnitude Attention-based Dynamic Pruningcited by 5

50 most cited of 83 All 83 (JSON)