mlsanityData to 14 Oct 2026

MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation

arXiv 2105.05912 ↗First posted 12 May 2021cs.CL cs.LGcited by 27

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

The advent of large pre-trained language models has given rise to rapid progress in the field of Natural Language Processing (NLP). While the performance of these models on standard benchmarks has scaled with size, compression techniques such as knowledge distillation have been key in making them practical. We present, MATE-KD, a novel text-based adversarial training algorithm which improves the performance of knowledge distillation. MATE-KD first trains a masked language model based generator to perturb text by maximizing the divergence between teacher and student logits. Then using knowledge distillation a student is trained on both the original and the perturbed training samples. We evaluate our algorithm, using BERT-based models, on the GLUE benchmark and demonstrate that MATE-KD outperforms competitive adversarial learning and data augmentation baselines. On the GLUE test set our 6 layer RoBERTa based model outperforms BERT-Large.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 35 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

30 of 35 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
18 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  3. 2017Mixed Precision Trainingcited by 1,156
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  6. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  10. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  11. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  12. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  13. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  14. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 106
  15. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  16. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  17. 2021Annealing Knowledge Distillationcited by 61
  18. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 9; 2023: 5; 2024: 3; 2025: 2.

27 papers.

Sort:

By area:

  1. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  2. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  3. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  4. 2021Kronecker Decomposition for GPT Compressioncited by 28
  5. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
  6. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 17
  7. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  8. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  9. 2024Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language Modelscited by 10
  10. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averagingcited by 9
  11. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantizationcited by 8
  12. 2023Speeding up Resnet Architecture with Layers Targeted Low Rank Decompositioncited by 8
15 more
  1. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  2. 2023Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentationcited by 7
  3. 2022When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentationcited by 5
  4. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5
  5. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4
  6. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4
  7. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  8. 2022Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimizationcited by 3
  9. 2022Kronecker Decomposition for Knowledge Graph Embeddingscited by 2
  10. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  11. 2025Evaluating Large Language Models for Anxiety, Depression, and Stress Detection: Insights into Prompting Strategies and Synthetic Datacited by 2
  12. 2021Knowledge Distillation with Noisy Labels for Natural Language Understandingcited by 1
  13. 2022Teacher Guided Training: An Efficient Framework for Knowledge Transfercited by 1
  14. 2023HARD: Hard Augmentations for Robust Distillationcited by 1
  15. 2024Accelerating the Low-Rank Decomposed Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
  2. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  3. 2021RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models
  4. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training
  5. 2024MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
  6. 2022On the benefits of knowledge distillation for adversarial robustness
  7. 2021KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
  8. 2023VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks