mlsanityData to 7 Oct 2026

Picking Winning Tickets Before Training by Preserving Gradient Flow

arXiv 2002.07376 ↗First posted 18 Feb 2020cs.LG cs.CV stat.MLcited by 467

Not on the map. Why?

Abstract (arXiv)

Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.

Key facts. Cited by 467 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 81; 2022: 92; 2023: 87; 2024: 90; 2025: 73; 2026: 44.

467 papers.

Sort:

By area:

  1. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  3. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  4. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  5. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  6. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  7. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  8. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  9. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  10. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  11. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  12. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
38 more
  1. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  2. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  3. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  4. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  5. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  6. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  7. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  8. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  9. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  10. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  11. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  12. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  13. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  14. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  15. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  16. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  17. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  18. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
  19. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  20. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  21. 2021Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Seriescited by 49
  22. 2021Meta-Learning Sparse Implicit Neural Representationscited by 48
  23. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  24. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  25. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  26. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  27. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  28. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  29. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  30. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  31. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  32. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41
  33. 2022Structural Pruning via Latency-Saliency Knapsackcited by 40
  34. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  35. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  36. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  37. 2023A Survey on Efficient Training of Transformerscited by 37
  38. 2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36

50 most cited of 467 All 467 (JSON)