mlsanityData to 7 Oct 2026

Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training

arXiv 2204.11218 ↗First posted 24 Apr 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent studies on the lottery ticket hypothesis (LTH) show that pre-trained language models (PLMs) like BERT contain matching subnetworks that have similar transfer learning performance as the original PLM. These subnetworks are found using magnitude-based pruning. In this paper, we find that the BERT subnetworks have even more potential than these studies have shown. Firstly, we discover that the success of magnitude pruning can be attributed to the preserved pre-training performance, which correlates with the downstream transferability. Inspired by this, we propose to directly optimize the subnetwork structure towards the pre-training objectives, which can better preserve the pre-training performance. Specifically, we train binary masks over model weights on the pre-training tasks, with the aim of preserving the universal transferability of the subnetwork, which is agnostic to any specific downstream tasks. We then fine-tune the subnetworks on the GLUE benchmark and the SQuAD dataset. The results show that, compared with magnitude pruning, mask training can effectively find BERT subnetworks with improved overall performance on downstream tasks. Moreover, our method is also more efficient in searching subnetworks and more advantageous when fine-tuning within a certain range of data scarcity. Our code is available at https://github.com/llyx97/TAMT.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 24 Apr 2022.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
27 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  11. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  12. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  13. 2019Structured Pruning of Large Language Modelscited by 201
  14. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  15. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  16. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  17. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  18. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  19. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  20. 2019Fast Sparse ConvNetscited by 86
  21. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  22. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  23. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  24. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  25. 2019Stabilizing the Lottery Ticket Hypothesiscited by 36
  26. 2020Pruning Convolutional Neural Networks with Self-Supervisioncited by 7
  27. 2021Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillationcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 4.

10 papers.

  1. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  2. 2024Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruningcited by 15
  3. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  4. 2024Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNNcited by 10
  5. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  6. 2023Adversarial Erasing with Pruned Elements: Towards Better Graph Lottery Ticketcited by 5
  7. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  8. 2023Uncovering the Hidden Cost of Model Compressioncited by 3
  9. 2023Towards Compute-Optimal Transfer Learningcited by 2
  10. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answeringcited by 0