Data to 7 Oct 2026

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

arXiv 2007.12223 ↗First posted 23 Jul 2020cs.LG cs.CL cs.NEcited by 275

Not on the map. Why?

Abstract (arXiv)

In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. In this work, we combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context. Codes available at https://github.com/VITA-Group/BERT-Tickets.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 66; 2023: 73; 2024: 42; 2025: 21; 2026: 16.

275 papers.

Sort:

By area:

2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
2022Structured Pruning Learns Compact and Accurate Modelscited by 167
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141

38 more

2021Learned Token Pruning for Transformerscited by 138
2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
2023Modular Deep Learningcited by 82
2022Unified Visual Transformer Compressioncited by 79
2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
2024A Complete Survey on LLM-based AI Chatbotscited by 71
2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
2022Advancing Model Pruning via Bi-level Optimizationcited by 55
2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
2021A Survey on Green Deep Learningcited by 52
2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
2021How Well Do Sparse Imagenet Models Transfer?cited by 50
2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
2021Playing Lottery Tickets with Vision and Languagecited by 45
2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
2021Recent Advances on Neural Network Pruning at Initializationcited by 41
2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40

50 most cited of 275 All 275 (JSON)