# Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?

arXiv: https://arxiv.org/abs/2102.11068 · HTML: https://mlsanity.com/paper/2102.11068/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 24 of 24)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection](https://mlsanity.com/paper/2003.01794/) (2020)
- [NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm](https://mlsanity.com/paper/1711.02017/) (2017)
- [AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates](https://mlsanity.com/paper/1907.03141/) (2019)
- [ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1812.11677/) (2018)
- [2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy](https://mlsanity.com/paper/1809.02220/) (2018)
- [Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1802.05747/) (2018)

## Built on by (most cited first, 22 of 22)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [Effective Model Sparsification by Scheduled Grow-and-Prune Methods](https://mlsanity.com/paper/2106.09857/) (2021)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [On the Effect of Pruning on Adversarial Robustness](https://mlsanity.com/paper/2108.04890/) (2021)
- [Rethinking Graph Lottery Tickets: Graph Sparsity Matters](https://mlsanity.com/paper/2305.02190/) (2023)
- [Most Activation Functions Can Win the Lottery Without Excessive Depth](https://mlsanity.com/paper/2205.02321/) (2022)
- [Convolutional and Residual Networks Provably Contain Lottery Tickets](https://mlsanity.com/paper/2205.02343/) (2022)
- [Masks, Signs, And Learning Rate Rewinding](https://mlsanity.com/paper/2402.19262/) (2024)
- [Plant 'n' Seek: Can You Find the Winning Ticket?](https://mlsanity.com/paper/2111.11153/) (2021)
- [Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training](https://mlsanity.com/paper/2209.11204/) (2022)
- [Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective](https://mlsanity.com/paper/2205.07320/) (2022)
- [Accelerating ViT Inference on FPGA through Static and Dynamic Pruning](https://mlsanity.com/paper/2403.14047/) (2024)
- [Lottery Tickets with Nonzero Biases](https://mlsanity.com/paper/2110.11150/) (2021)
- [COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks](https://mlsanity.com/paper/2212.12770/) (2022)
- [When Layers Play the Lottery, all Tickets Win at Initialization](https://mlsanity.com/paper/2301.10835/) (2023)
- [Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes](https://mlsanity.com/paper/2405.05808/) (2024)
- [PrUE: Distilling Knowledge from Sparse Teacher Networks](https://mlsanity.com/paper/2207.00586/) (2022)
- [Privacy Amplification by Iteration for ADMM with (Strongly) Convex Objective Functions](https://mlsanity.com/paper/2312.08685/) (2023)
- [AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking](https://mlsanity.com/paper/2409.17728/) (2024)

## Abstract (arXiv)

In deep model compression, the recent finding "Lottery Ticket Hypothesis" (LTH) (Frankle & Carbin, 2018) pointed out that there could exist a winning ticket (i.e., a properly pruned sub-network together with original weight initialization) that can achieve competitive performance than the original dense network. However, it is not easy to observe such winning property in many scenarios, where for example, a relatively large learning rate is used even if it benefits training the original dense model. In this work, we investigate the underlying condition and rationale behind the winning property, and find that the underlying reason is largely attributed to the correlation between initialized weights and final-trained weights when the learning rate is not sufficiently large. Thus, the existence of winning property is correlated with an insufficient DNN pretraining, and is unlikely to occur for a well-trained DNN. To overcome this limitation, we propose the "pruning & fine-tuning" method that consistently outperforms lottery ticket sparse training under the same pruning algorithm and the same total training epochs. Extensive experiments over multiple deep models (VGG, ResNet, MobileNet-v2) on different datasets have been conducted to justify our proposals.