# Dual Lottery Ticket Hypothesis

arXiv: https://arxiv.org/abs/2203.04248 · HTML: https://mlsanity.com/paper/2203.04248/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization](https://mlsanity.com/paper/1902.05967/) (2019)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [Deep Rewiring: Training very sparse deep networks](https://mlsanity.com/paper/1711.05136/) (2017)
- [Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers](https://mlsanity.com/paper/1802.00124/) (2018)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [Neural Pruning via Growing Regularization](https://mlsanity.com/paper/2012.09243/) (2020)
- [A Signal Propagation Perspective for Pruning Neural Networks at Initialization](https://mlsanity.com/paper/1906.06307/) (2019)
- [Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks](https://mlsanity.com/paper/1905.10961/) (2019)
- [WoodFisher: Efficient Second-Order Approximation for Neural Network Compression](https://mlsanity.com/paper/2004.14340/) (2020)
- [Structured Pruning for Efficient ConvNets via Incremental Regularization](https://mlsanity.com/paper/1811.08390/) (2018)
- [Generalized Dropout](https://mlsanity.com/paper/1611.06791/) (2016)

## Built on by (most cited first, 27 of 27)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques](https://mlsanity.com/paper/2406.02500/) (2024)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [FedMef: Towards Memory-efficient Federated Dynamic Pruning](https://mlsanity.com/paper/2403.14737/) (2024)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [The Snowflake Hypothesis: Training Deep GNN with One Node One Receptive field](https://mlsanity.com/paper/2308.10051/) (2023)
- [Parameter-Efficient Masking Networks](https://mlsanity.com/paper/2210.06699/) (2022)
- [Data Level Lottery Ticket Hypothesis for Vision Transformers](https://mlsanity.com/paper/2211.01484/) (2022)
- [Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized?](https://mlsanity.com/paper/2505.10679/) (2025)
- [Boosting Large Language Models with Mask Fine-Tuning](https://mlsanity.com/paper/2503.22764/) (2025)
- [Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution](https://mlsanity.com/paper/2303.09650/) (2023)
- [Enhanced Sparsification via Stimulative Training](https://mlsanity.com/paper/2403.06417/) (2024)
- [Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance](https://mlsanity.com/paper/2410.16151/) (2024)
- [SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks](https://mlsanity.com/paper/2510.08513/) (2025)
- [One-Shot Pruning for Fast-adapting Pre-trained Models on Devices](https://mlsanity.com/paper/2307.04365/) (2023)
- [One-Cycle Structured Pruning via Stability-Driven Subnetwork Search](https://mlsanity.com/paper/2501.13439/) (2025)
- [Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks](https://mlsanity.com/paper/2301.11560/) (2023)
- [Learning to Learn with Indispensable Connections](https://mlsanity.com/paper/2304.02862/) (2023)
- [Surrogate Lagrangian Relaxation: A Path To Retrain-free Deep Neural Network Pruning](https://mlsanity.com/paper/2304.04120/) (2023)
- [Learning Sparse Neural Networks with Identity Layers](https://mlsanity.com/paper/2307.07389/) (2023)
- [Aggressive or Imperceptible, or Both: Network Pruning Assisted Hybrid Byzantines in Federated Learning](https://mlsanity.com/paper/2404.06230/) (2024)
- [S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning](https://mlsanity.com/paper/2410.07046/) (2024)
- [NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling](https://mlsanity.com/paper/2505.17909/) (2025)
- [SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions](https://mlsanity.com/paper/2510.08999/) (2025)

## Abstract (arXiv)

Fully exploiting the learning capacity of neural networks requires overparameterized dense networks. On the other side, directly training sparse neural networks typically results in unsatisfactory performance. Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. Concretely, it claims there exist winning tickets from a randomly initialized network found by iterative magnitude pruning and preserving promising trainability (or we say being in trainable condition). In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark, then go from a complementary direction to articulate the Dual Lottery Ticket Hypothesis (DLTH): Randomly selected subnetworks from a randomly initialized dense network can be transformed into a trainable condition and achieve admirable performance compared with LTH -- random tickets in a given lottery pool can be transformed into winning tickets. Specifically, by using uniform-randomly selected subnetworks to represent the general cases, we propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH. Concretely, we introduce a regularization term to borrow learning capacity and realize information extrusion from the weights which will be masked. After finishing the transformation for the randomly selected subnetworks, we conduct the regular finetuning to evaluate the model using fair comparisons with LTH and other strong baselines. Extensive experiments on several public datasets and comparisons with competitive approaches validate our DLTH as well as the effectiveness of the proposed model RST. Our work is expected to pave a way for inspiring new research directions of sparse network training in the future. Our code is available at https://github.com/yueb17/DLTH.