mlsanityData to 7 Oct 2026

Knowledge Distillation with Noisy Labels for Natural Language Understanding

arXiv 2109.10147 ↗First posted 21 Sep 2021cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications. However, one neglected area of research is the impact of noisy (corrupted) labels on KD. We present, to the best of our knowledge, the first study on KD with noisy labels in Natural Language Understanding (NLU). We document the scope of the problem and present two methods to mitigate the impact of label noise. Experiments on the GLUE benchmark show that our methods are effective even under high noise levels. Nevertheless, our results indicate that more research is necessary to cope with label noise under the KD.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

38 of 51 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  12. 2017A Closer Look at Memorization in Deep Networkscited by 936
26 more
  1. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  2. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  3. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  4. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  5. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  6. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  7. 2018Learning to Learn from Noisy Labeled Datacited by 160
  8. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 121
  9. 2019Combating Label Noise in Deep Learning Using Abstentioncited by 86
  10. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  11. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  12. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  13. 2021Annealing Knowledge Distillationcited by 61
  14. 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 27
  15. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  16. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  17. 2021End-to-End Self-Debiasing Framework for Robust NLU Trainingcited by 24
  18. 2019An Effective Label Noise Model for DNN Text Classificationcited by 22
  19. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  20. 2017Fidelity-Weighted Learningcited by 19
  21. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  22. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  23. 2021Towards Robustness to Label Noise in Text Classification via Noise Modelingcited by 9
  24. 2020Role-Wise Data Augmentation for Knowledge Distillationcited by 8
  25. 2020Learning to Reweight with Deep Interactionscited by 4
  26. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FedSIR: Spectral Client Identification and Relabeling for Federated Learning with Noisy Labelscited by 1