# One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

arXiv: https://arxiv.org/abs/1906.02773 · HTML: https://mlsanity.com/paper/1906.02773/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 30 of 102)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models](https://mlsanity.com/paper/2112.00029/) (2021)
- [The State of Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2206.10369/) (2022)
- [Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization](https://mlsanity.com/paper/2105.12002/) (2021)
- [Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?](https://mlsanity.com/paper/2107.00166/) (2021)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Meta-Learning Sparse Implicit Neural Representations](https://mlsanity.com/paper/2110.14678/) (2021)
- [Efficient Lottery Ticket Finding: Less Data is More](https://mlsanity.com/paper/2106.03225/) (2021)
- [Recent Advances on Neural Network Pruning at Initialization](https://mlsanity.com/paper/2103.06460/) (2021)
- [The Tunnel Effect: Building Data Representations in Deep Neural Networks](https://mlsanity.com/paper/2305.19753/) (2023)
- [GANs Can Play Lottery Tickets Too](https://mlsanity.com/paper/2106.00134/) (2021)
- [On the Existence of Universal Lottery Tickets](https://mlsanity.com/paper/2111.11146/) (2021)
- [XPrompt: Exploring the Extreme of Prompt Tuning](https://mlsanity.com/paper/2210.04457/) (2022)
- [Discovering Language-neutral Sub-networks in Multilingual Language Models](https://mlsanity.com/paper/2205.12672/) (2022)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)
- [On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.01648/) (2021)
- [Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction](https://mlsanity.com/paper/2310.08670/) (2023)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Spectral Graph Pruning Against Over-Squashing and Over-Smoothing](https://mlsanity.com/paper/2404.04612/) (2024)
- [The Elastic Lottery Ticket Hypothesis](https://mlsanity.com/paper/2103.16547/) (2021)
- [Rethinking Graph Lottery Tickets: Graph Sparsity Matters](https://mlsanity.com/paper/2305.02190/) (2023)
- [Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?](https://mlsanity.com/paper/2102.11068/) (2021)
- [Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution](https://mlsanity.com/paper/2104.13424/) (2021)
- [Pruning Randomly Initialized Neural Networks with Iterative Randomization](https://mlsanity.com/paper/2106.09269/) (2021)
- [Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning](https://mlsanity.com/paper/2206.01342/) (2022)
- [Knowledge Evolution in Neural Networks](https://mlsanity.com/paper/2103.05152/) (2021)
- [LPViT: Low-Power Semi-structured Pruning for Vision Transformers](https://mlsanity.com/paper/2407.02068/) (2024)
- [No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models](https://mlsanity.com/paper/2202.02664/) (2022)

## Abstract (arXiv)

The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same winning tickets across a variety of datasets and optimizers. However, the generality of winning ticket initializations remains unclear. Here, we attempt to answer this question by generating winning tickets for one training configuration (optimizer and dataset) and evaluating their performance on another configuration. Perhaps surprisingly, we found that, within the natural images domain, winning ticket initializations generalized across a variety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, and Places365, often achieving performance close to that of winning tickets generated on the same dataset. Moreover, winning tickets generated using larger datasets consistently transferred better than those generated using smaller datasets. We also found that winning ticket initializations generalize across optimizers with high performance. These results suggest that winning ticket initializations generated by sufficiently large datasets contain inductive biases generic to neural networks more broadly which improve training across many settings and provide hope for the development of better initialization methods.