Data to 14 Oct 2026

Rigging the Lottery: Making All Tickets Winners

arXiv 1911.11134 ↗First posted 25 Nov 2019cs.LG cs.CV stat.MLcited by 495

Not on the map. Why?

Abstract (arXiv)

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static. Code used in our work can be found in github.com/google-research/rigl.

Key facts. Cited by 495 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 71; 2022: 80; 2023: 96; 2024: 103; 2025: 77; 2026: 68.

495 papers.

Sort:

By area:

2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
2021Accelerating Sparse Deep Neural Networkscited by 223
2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
2022A Review of Sparse Expert Models in Deep Learningcited by 148

38 more

2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
2021The Efficiency Misnomercited by 98
2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
2021Top-KAST: Top-K Always Sparse Trainingcited by 79
2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 78
2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
2022Advancing Model Pruning via Bi-level Optimizationcited by 56
2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 55
2022SparCL: Sparse Continual Learning on the Edgecited by 55
2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
2021How Well Do Sparse Imagenet Models Transfer?cited by 50
2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
2025Weight-sparse transformers have interpretable circuitscited by 44
2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43
2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42

50 most cited of 495 All 495 (JSON)