mlsanityData to 7 Oct 2026

How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding

arXiv 2109.05696 ↗First posted 13 Sep 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing (NLP) applications, little is understood about how one KD algorithm compares to another and whether these approaches can be complimentary to each other. In this work, we evaluate various KD algorithms on in-domain, out-of-domain and adversarial testing. We propose a framework to assess the adversarial robustness of multiple KD algorithms. Moreover, we introduce a new KD algorithm, Combined-KD, which takes advantage of two promising approaches (better training scheme and more efficient data augmentation). Our extensive experimental results show that Combined-KD achieves state-of-the-art results on the GLUE benchmark, out-of-domain generalization, and adversarial robustness compared to competitive methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

36 of 46 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2016Adversarial examples in the physical worldcited by 1,762
24 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  5. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  7. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  8. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  10. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  11. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  12. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  13. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  14. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  15. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  16. 2021Annealing Knowledge Distillationcited by 61
  17. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  18. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  19. 2021End-to-End Self-Debiasing Framework for Robust NLU Trainingcited by 24
  20. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  21. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  22. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  23. 2020Role-Wise Data Augmentation for Knowledge Distillationcited by 8
  24. 2019Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inferencecited by 6

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 1.

4 papers.

  1. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  2. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  3. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4
  4. 2021Knowledge Distillation with Noisy Labels for Natural Language Understandingcited by 1