mlsanityData to 7 Oct 2026

Robust Lottery Tickets for Pre-trained Language Models

arXiv 2211.03013 ↗First posted 6 Nov 2022cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks(winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be notrobust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs. Since the loss is not differentiable for the binary mask, we assign the hard concrete distribution to the masks and encourage their sparsity using a smoothing approximation of L0 regularization.Furthermore, we design an adversarial loss objective to guide the search for robust tickets and ensure that the tickets perform well bothin accuracy and robustness. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 6 Nov 2022.

Builds on

38 of 53 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2018Robustness May Be at Odds with Accuracycited by 873
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
26 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 803
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  4. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  5. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  6. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  7. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  8. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  9. 2019Revealing the Dark Secrets of BERTcited by 300
  10. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  11. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  12. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  13. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  14. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  15. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  16. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  17. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  18. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  19. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  20. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  21. 2019Model Compression with Adversarial Robustness: A Unified Optimization Frameworkcited by 71
  22. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  23. 2021Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitutioncited by 55
  24. 2020Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuningcited by 39
  25. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  26. 2019Towards Compact and Robust Deep Neural Networkscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 4; 2026: 7.

16 papers.

Sort:

By area:

  1. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
  2. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  3. 2023Successfully Applying Lottery Ticket Hypothesis to Diffusion Modelcited by 11
  4. 2025StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matchingcited by 8
  5. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  6. 2022Efficient Adversarial Training with Robust Early-Bird Ticketscited by 5
  7. 2026DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuningcited by 5
  8. 2026Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levelscited by 5
  9. 2024KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language Modelscited by 3
  10. 2026Reinforcement Learning Enhanced Multi-hop Reasoning for Temporal Knowledge Question Answeringcited by 2
  11. 2025Using External knowledge to Enhanced PLM for Semantic Matchingcited by 0
  12. 2025Boosting Neural Language Inference via Cascaded Interactive Reasoningcited by 0
4 more
  1. 2025NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensemblingcited by 0
  2. 2026Temporal-Aware Heterogeneous Graph Reasoning with Multi-View Fusion for Temporal Question Answeringcited by 0
  3. 2026Efficient Task Adaptation in Large Language Models via Selective Parameter Optimizationcited by 0
  4. 2026Multi-Granularity Reasoning for Natural Language Inferencecited by 0