mlsanityData to 7 Oct 2026

Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Tickets

arXiv 2207.09638 ↗First posted 20 Jul 2022cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Over-parameterized models, typically pretrained language models (LMs), have shown an appealing expressive power due to their small learning bias. However, the huge learning capacity of LMs can also lead to large learning variance. In a pilot study, we find that, when faced with multiple domains, a critical portion of parameters behave unexpectedly in a domain-specific manner while others behave in a domain-general one. Motivated by this phenomenon, we for the first time posit that domain-general parameters can underpin a domain-general LM that can be derived from the original LM. To uncover the domain-general LM, we propose to identify domain-general parameters by playing lottery tickets (dubbed doge tickets). In order to intervene the lottery, we propose a domain-general score, which depicts how domain-invariant a parameter is by associating it with the variance. Comprehensive experiments are conducted on the Amazon, Mnli and OntoNotes datasets. The results show that the doge tickets obtains an improved out-of-domain generalization in comparison with a range of competitive baselines. Analysis results further hint the existence of domain-general parameters and the performance consistency of doge tickets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 20 Jul 2022.

Builds on

28 of 33 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2019Invariant Risk Minimizationcited by 1,832
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
16 more
  1. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  4. 2019The State of Sparsity in Deep Neural Networkscited by 471
  5. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  6. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  7. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  10. 2016Variance-based regularization with convex objectivescited by 153
  11. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  12. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  13. 2013Deep Learning of Representations: Looking Forwardcited by 84
  14. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  15. 2016Faster CNNs with Direct Sparse Convolutions and Guided Pruningcited by 39
  16. 2020Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecturecited by 23

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2.

4 papers.

  1. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
  2. 2022Sparse Teachers Can Be Dense with Knowledgecited by 11
  3. 2023Sparse Contrastive Learning of Sentence Embeddingscited by 0
  4. 2023On Elastic Language Modelscited by 0