mlsanityData to 7 Oct 2026

Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization

arXiv 2105.12002 ↗First posted 25 May 2021cs.LG cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match the performance of the full model. In this paper, we study such a collection of tickets, which is referred to as ``winning tickets'', in extremely over-parametrized models, e.g., pre-trained language models. We observe that at certain compression ratios, the generalization performance of the winning tickets can not only match but also exceed that of the full model. In particular, we observe a phase transition phenomenon: As the compression ratio increases, generalization performance of the winning tickets first improves then deteriorates after a certain threshold. We refer to the tickets on the threshold as ``super tickets''. We further show that the phase transition is task and model dependent -- as the model size becomes larger and the training data set becomes smaller, the transition becomes more pronounced. Our experiments on the GLUE benchmark show that the super tickets improve single task fine-tuning by 0.9 points on BERT-base and 1.0 points on BERT-large, in terms of task-average score. We also demonstrate that adaptively sharing the super tickets across tasks benefits multi-task learning.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

35 of 52 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2018Neural Network Acceptability Judgmentscited by 952
23 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2018Rethinking the Value of Network Pruningcited by 595
  4. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  5. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  6. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  7. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  8. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  9. 2019Structured Pruning of Large Language Modelscited by 201
  10. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  11. 2020The Early Phase of Neural Network Trainingcited by 114
  12. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  13. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  14. 2019Winning the Lottery with Continuous Sparsificationcited by 99
  15. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  16. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  17. 2020The Lottery Ticket Hypothesis for Object Recognitioncited by 34
  18. 2019Learning Sparse Sharing Architectures for Multiple Taskscited by 31
  19. 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
  20. 2020Scheduled DropHead: A Regularization Method for Transformer Modelscited by 28
  21. 2019Evaluating Lottery Tickets Under Distributional Shiftscited by 23
  22. 2020Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecturecited by 23
  23. 2020Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 17; 2023: 15; 2024: 12; 2025: 6; 2026: 2.

56 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  5. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  7. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  8. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  9. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  10. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  11. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  12. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
38 more
  1. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  2. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  3. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  4. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  5. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  6. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  7. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  8. 2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 18
  9. 2024TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidationcited by 18
  10. 2022Robust Lottery Tickets for Pre-trained Language Modelscited by 16
  11. 2022No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Modelscited by 15
  12. 2025Provable Separations between Memorization and Generalization in Diffusion Modelscited by 14
  13. 2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12
  14. 2022Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Trainingcited by 10
  15. 2022Gradient-based Intra-attention Pruning on Pre-trained Language Modelscited by 9
  16. 2023Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learningcited by 9
  17. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Modelscited by 9
  18. 2022Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Modelcited by 7
  19. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  20. 2022Pruning Pre-trained Language Models Without Fine-Tuningcited by 7
  21. 2023Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Modelscited by 7
  22. 2022MiniDisc: Minimal Distillation Schedule for Language Model Compressioncited by 5
  23. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Modelscited by 5
  24. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  25. 2025Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generationcited by 5
  26. 2022Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Ticketscited by 4
  27. 2022AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuningcited by 4
  28. 2024Graph-Structured Speculative Decodingcited by 4
  29. 2022Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiencycited by 3
  30. 2023Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compressioncited by 3
  31. 2024VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruningcited by 3
  32. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 3
  33. 2023Pruning Pre-trained Language Models with Principled Importance and Self-regularizationcited by 2
  34. 2024Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysiscited by 2
  35. 2023Improving Multitask Retrieval by Promoting Task Specializationcited by 1
  36. 2024FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Networkcited by 1
  37. 2024Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Priorcited by 1
  38. 2025SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitializationcited by 1

50 most cited of 56 All 56 (JSON)