mlsanityData to 7 Oct 2026

Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

arXiv 2205.11833 ↗First posted 24 May 2022cs.LG cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

30 of 48 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2020What is the State of Neural Network Pruning?cited by 578
  12. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
18 more
  1. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  2. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  3. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  4. 2020Training independent subnetworks for robust predictioncited by 177
  5. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  6. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  7. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  8. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  9. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  10. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  11. 2020Masksembles for Uncertainty Estimationcited by 88
  12. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  13. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  14. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  15. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  16. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  17. 2018DESlib: A Dynamic ensemble selection library in Pythoncited by 18
  18. 2021Ex uno plures: Splitting One Model into an Ensemble of Subnetworkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 1; 2026: 1.

7 papers.

  1. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  2. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  3. 2022Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewardscited by 14
  4. 2025A survey of diversity quantification in natural language processing: The why, what, where and howcited by 4
  5. 2023ASPEST: Bridging the Gap Between Active Learning and Selective Predictioncited by 3
  6. 2024Sharpness-diversity tradeoff: improving flat ensembles with SharpBalancecited by 3
  7. 2026Virtual neural networks: hundreds of souls in a bodycited by 0