mlsanityData to 7 Oct 2026

When BERT Plays the Lottery, All Tickets Are Winning

arXiv 2005.00561 ↗First posted 1 May 2020cs.CL cs.LGcited by 114

Not on the map. Why?

Abstract (arXiv)

Large Transformer-based models were shown to be reducible to a smaller number of self-attention heads and layers. We consider this phenomenon from the perspective of the lottery ticket hypothesis, using both structured and magnitude pruning. For fine-tuned BERT, we show that (a) it is possible to find subnetworks achieving performance that is comparable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. Strikingly, with structured pruning even the worst possible subnetworks remain highly trainable, indicating that most pre-trained BERT weights are potentially useful. We also study the "good" subnetworks to see if their success can be attributed to superior linguistic knowledge, but find them unstable, and not explained by meaningful self-attention patterns.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 26; 2023: 31; 2024: 13; 2025: 11; 2026: 3.

114 papers.

Sort:

By area:

  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  6. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  7. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  8. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  9. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  10. 2021Learned Token Pruning for Transformerscited by 138
  11. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  12. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
38 more
  1. 2023Modular Deep Learningcited by 82
  2. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  5. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  6. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  7. 2021A Survey on Green Deep Learningcited by 52
  8. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  9. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  10. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  11. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  12. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  13. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  14. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  15. 2021Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networkscited by 29
  16. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  17. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  18. 2021Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimizationcited by 27
  19. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  20. 2021Pruning Self-attentions into Convolutional Layers in Single Pathcited by 24
  21. 2023Rethinking Graph Lottery Tickets: Graph Sparsity Matterscited by 23
  22. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  23. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  24. 2023Fairness-Aware Structured Pruning in Transformerscited by 21
  25. 2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
  26. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  27. 2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 18
  28. 2022Can Model Compression Improve NLP Fairnesscited by 18
  29. 2022Robust Lottery Tickets for Pre-trained Language Modelscited by 16
  30. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  31. 2024Superficial Safety Alignment Hypothesiscited by 14
  32. 2022An Automatic and Efficient BERT Pruning for Edge AI Systemscited by 13
  33. 2023Examining Modularity in Multilingual LMs via Language-Specialized Subnetworkscited by 13
  34. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  35. 2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12
  36. 2022Sparse Teachers Can Be Dense with Knowledgecited by 11
  37. 2023Successfully Applying Lottery Ticket Hypothesis to Diffusion Modelcited by 11
  38. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11

50 most cited of 114 All 114 (JSON)