mlsanityData to 7 Oct 2026

Data Level Lottery Ticket Hypothesis for Vision Transformers

arXiv 2211.01484 ↗First posted 2 Nov 2022cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The conventional lottery ticket hypothesis (LTH) claims that there exists a sparse subnetwork within a dense neural network and a proper random initialization method called the winning ticket, such that it can be trained from scratch to almost as good as the dense counterpart. Meanwhile, the research of LTH in vision transformers (ViTs) is scarcely evaluated. In this paper, we first show that the conventional winning ticket is hard to find at the weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs to input data consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches. We call this subset of input patches the em winning tickets, which represent a significant amount of information in the input data. We use a ticket selector to generate the winning tickets based on the informativeness of patches for various types of ViT, including DeiT, LV-ViT, and Swin Transformers. The experiments show that there is a clear difference between the performance of models trained with winning tickets and randomly selected subsets, which verifies our proposed theory. We elaborate on the analogical similarity between our proposed Data-LTH-ViTs and the conventional LTH to further verify the integrity of our theory. The Source codes are available at https://github.com/shawnricecake/vit-lottery-ticket-input.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

37 of 51 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  11. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  12. 2019The State of Sparsity in Deep Neural Networkscited by 471
25 more
  1. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  2. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  3. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  4. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  5. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  6. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  7. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  8. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  9. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  10. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  11. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  12. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  13. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  14. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  15. 2022Unified Visual Transformer Compressioncited by 79
  16. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  17. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  18. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
  19. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  20. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  21. 2020The Lottery Ticket Hypothesis for Object Recognitioncited by 34
  22. 2022Dual Lottery Ticket Hypothesiscited by 27
  23. 2019Evaluating Lottery Tickets Under Distributional Shiftscited by 23
  24. 2021Multi-Dimensional Model Compression of Vision Transformercited by 13
  25. 2021MIA-Former: Efficient and Robust Vision Transformers via Multi-grained Input-Adaptationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 1.

7 papers.

  1. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 35
  2. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  3. 2024Numerical Pruning for Efficient Autoregressive Modelscited by 24
  4. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  5. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  6. 2023Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Networkcited by 1
  7. 2026Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gatescited by 0