mlsanityData to 7 Oct 2026

Comparing Rewinding and Fine-tuning in Neural Network Pruning

arXiv 2003.02389 ↗First posted 5 Mar 2020cs.LG stat.MLcited by 238

Not on the map. Why?

Abstract (arXiv)

Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, we compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al., (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.

Key facts. Cited by 238 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 59; 2023: 47; 2024: 36; 2025: 25; 2026: 14.

238 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  2. 2023DepGraph: Towards Any Structural Pruningcited by 202
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  5. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  6. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  7. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  8. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  9. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  10. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  11. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  12. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
38 more
  1. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  2. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  3. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 61
  4. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  5. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  6. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  7. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  8. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  9. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  10. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  11. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  12. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  13. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  14. 2024A Survey on Transformer Compressioncited by 44
  15. 2022Pruning has a disparate impact on model accuracycited by 43
  16. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 42
  17. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  18. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  19. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  20. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  21. 2022Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradientscited by 34
  22. 2022Rare Gems: Finding Lottery Tickets at Initializationcited by 34
  23. 2021GANs Can Play Lottery Tickets Toocited by 32
  24. 2023OTOV2: Automatic, Generic, User-Friendlycited by 32
  25. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  26. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  27. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
  28. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  29. 2021Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networkscited by 29
  30. 2022Coarsening the Granularity: Towards Structurally Sparse Lottery Ticketscited by 27
  31. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  32. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  33. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  34. 2024Federated LoRA with Sparse Communicationcited by 26
  35. 2021Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspectivecited by 25
  36. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  37. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  38. 2023Pruning Deep Neural Networks from a Sparsity Perspectivecited by 25

50 most cited of 238 All 238 (JSON)