mlsanityData to 7 Oct 2026

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

arXiv 1803.03635 ↗First posted 9 Mar 2018cs.LG cs.AI cs.NEcited by 2,250

Not on the map. Why?

Cited by 39 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Key facts. Cited by 2,250 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 337; 2022: 376; 2023: 437; 2024: 386; 2025: 387; 2026: 326.

2,250 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  10. 2021Going deeper with Image Transformerscited by 759
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
38 more
  1. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  2. 2021Neural Fields in Visual Computing and Beyondcited by 539
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  4. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  7. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  8. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  9. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  10. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  11. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  13. 2023A Survey on Model Compression for Large Language Modelscited by 263
  14. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  15. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  16. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  17. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  18. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  19. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  20. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  21. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  22. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  23. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  24. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  25. 2023Structural Pruning for Diffusion Modelscited by 193
  26. 2023EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingScited by 191
  27. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  28. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  29. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  30. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  31. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 166
  32. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  33. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  34. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  35. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  36. 2024How do Large Language Models Handle Multilingualism?cited by 148
  37. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  38. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142

50 most cited of 2,250 All 2,250 (JSON)