mlsanityData to 7 Oct 2026

Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets

arXiv 2202.04736 ↗First posted 9 Feb 2022cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i.e., winning tickets) that can be trained in isolation to match full accuracy. Despite many exciting efforts being made, there is one "commonsense" rarely challenged: a winning ticket is found by iterative magnitude pruning (IMP) and hence the resultant pruned subnetworks have only unstructured sparsity. That gap limits the appeal of winning tickets in practice, since the highly irregular sparse patterns are challenging to accelerate on hardware. Meanwhile, directly substituting structured pruning for unstructured pruning in IMP damages performance more severely and is usually unable to locate winning tickets. In this paper, we demonstrate the first positive result that a structurally sparse winning ticket can be effectively found in general. The core idea is to append "post-processing techniques" after each round of (unstructured) IMP, to enforce the formation of structural sparsity. Specifically, we first "re-fill" pruned elements back in some channels deemed to be important, and then "re-group" non-zero elements to create flexible group-wise structural patterns. Both our identified channel- and group-wise structural subnetworks win the lottery, with substantial inference speedups readily supported by existing hardware. Extensive experiments, conducted on diverse datasets across multiple network backbones, consistently validate our proposal, showing that the hardware acceleration roadblock of LTH is now removed. Specifically, the structural winning tickets obtain up to {64.93%, 64.84%, 60.23%} running time savings at {36%~80%, 74%, 58%} sparsity on {CIFAR, Tiny-ImageNet, ImageNet}, while maintaining comparable accuracy. Code is at https://github.com/VITA-Group/Structure-LTH.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

52 of 70 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2016Wide Residual Networkscited by 3,057
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  11. 2016An overview of gradient descent optimization algorithmscited by 1,021
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
38 more
  1. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  2. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  5. 2020What is the State of Neural Network Pruning?cited by 578
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  8. 2019Importance Estimation for Neural Network Pruningcited by 531
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2019The State of Sparsity in Deep Neural Networkscited by 471
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  13. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  14. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  15. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  16. 2021Deep learning: a statistical viewpointcited by 239
  17. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  18. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  19. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  20. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  21. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
  22. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  23. 2020The Early Phase of Neural Network Trainingcited by 114
  24. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
  25. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  26. 2020Channel Pruning via Automatic Structure Searchcited by 84
  27. 2020PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruningcited by 75
  28. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  29. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  30. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  31. 2019PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devicescited by 48
  32. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  33. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  34. 2018ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multiplierscited by 42
  35. 2016Faster CNNs with Direct Sparse Convolutions and Guided Pruningcited by 39
  36. 2017Exploring the Regularity of Sparse Structure in Convolutional Neural Networkscited by 37
  37. 2019Optimizing Speech Recognition For The Edgecited by 32
  38. 2021GANs Can Play Lottery Tickets Toocited by 32

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 8; 2023: 9; 2024: 5; 2025: 5.

27 papers.

Sort:

By area:

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  3. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  4. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  5. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  6. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  7. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  8. 2023Rethinking Graph Lottery Tickets: Graph Sparsity Matterscited by 23
  9. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  10. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  11. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  12. 2022Towards Sparsification of Graph Neural Networkscited by 18
15 more
  1. 2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
  2. 2022Trainability Preserving Neural Pruningcited by 16
  3. 2022To update or not to update? Neurons at equilibrium in deep modelscited by 13
  4. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
  5. 2022Lottery Pools: Winning More by Interpolating Tickets without Increasing Training or Inference Costcited by 11
  6. 2023Automatic Attention Pruning: Improving and Automating Model Pruning using Attentionscited by 11
  7. 2023Successfully Applying Lottery Ticket Hypothesis to Diffusion Modelcited by 11
  8. 2025Not All Explanations for Deep Learning Phenomena Are Equally Valuablecited by 3
  9. 2025Uncovering Critical Features for Deepfake Detection through the Lottery Ticket Hypothesiscited by 3
  10. 2023Towards Compute-Optimal Transfer Learningcited by 2
  11. 2023Concept-Monitor: Understanding DNN training through individual neuronscited by 2
  12. 2024Enhanced Sparsification via Stimulative Trainingcited by 2
  13. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Modelscited by 2
  14. 2024S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruningcited by 0
  15. 2025C-SWAP: Explainability-Aware Structured Pruning for Efficient Neural Networks Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
  2. 2023The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initialization
  3. 2026Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs
  4. 2023The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks
  5. 2023SparseByteNN: A Novel Mobile Inference Acceleration Framework Based on Fine-Grained Group Sparsity
  6. 2026How Sparse Probability Maps Shape Mixture-of-Experts Routing
  7. 2024Is Oracle Pruning the True Oracle?
  8. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling