mlsanityData to 7 Oct 2026

Annealing Knowledge Distillation

arXiv 2104.07163 ↗First posted 14 Apr 2021cs.CL cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

Significant memory and computational requirements of large deep neural networks restrict their application on edge devices. Knowledge distillation (KD) is a prominent model compression technique for deep neural networks in which the knowledge of a trained large teacher model is transferred to a smaller student model. The success of knowledge distillation is mainly attributed to its training objective function, which exploits the soft-target information (also known as "dark knowledge") besides the given regular hard labels in a training set. However, it is shown in the literature that the larger the gap between the teacher and the student networks, the more difficult is their training using knowledge distillation. To address this shortcoming, we propose an improved knowledge distillation method (called Annealing-KD) by feeding the rich information provided by the teacher's soft-targets incrementally and more efficiently. Our Annealing-KD technique is based on a gradual transition over annealed soft-targets generated by the teacher at different temperatures in an iterative process, and therefore, the student is trained to follow the annealed teacher output in a step-by-step manner. This paper includes theoretical and empirical evidence as well as practical experiments to support the effectiveness of our Annealing-KD method. We did a comprehensive set of experiments on different tasks such as image classification (CIFAR-10 and 100) and NLP language inference with BERT-based models on the GLUE benchmark and consistently got superior results.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

26 of 39 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
14 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2017Deep Mutual Learningcited by 621
  4. 2015Listen, Attend and Spellcited by 542
  5. 2018Born Again Neural Networkscited by 539
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  9. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  10. 2015Unifying distillation and privileged informationcited by 218
  11. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  12. 2019YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detectioncited by 16
  13. 2018Tensor Decomposition for Compressing Recurrent Neural Networkcited by 10
  14. 2019Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decompositioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 12; 2023: 10; 2024: 10; 2025: 9; 2026: 8.

61 papers.

Sort:

By area:

  1. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  2. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  3. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  4. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  5. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  6. 2021Kronecker Decomposition for GPT Compressioncited by 27
  7. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  8. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  9. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  10. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  11. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  12. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
38 more
  1. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  2. 2022HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectorscited by 12
  3. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selectioncited by 12
  4. 2025LLM-Oriented Token-Adaptive Knowledge Distillationcited by 12
  5. 2023Towards Understanding and Improving Knowledge Distillation for Neural Machine Translationcited by 10
  6. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averagingcited by 9
  7. 2021Language Modelling via Learning to Rankcited by 7
  8. 2023Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentationcited by 7
  9. 2023Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Modelscited by 6
  10. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  11. 2024Dynamic Temperature Knowledge Distillationcited by 6
  12. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5
  13. 2024CTSM: Combining Trait and State Emotions for Empathetic Response Modelcited by 5
  14. 2025Semantic Communications with Computer Vision Sensing for Edge Video Transmissioncited by 5
  15. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4
  16. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4
  17. 2024Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillationcited by 4
  18. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  19. 2021iRNN: Integer-only Recurrent Neural Networkcited by 3
  20. 2022Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimizationcited by 3
  21. 2024AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weightingcited by 3
  22. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 3
  23. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  24. 2021Learning to Teach with Student Feedbackcited by 2
  25. 2021Attention-Free Keyword Spottingcited by 2
  26. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  27. 2024EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Modelscited by 2
  28. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  29. 2021Knowledge Distillation with Noisy Labels for Natural Language Understandingcited by 1
  30. 2022VEM2L: A Plug-and-play Framework for Fusing Text and Structure Knowledge on Sparse Knowledge Graph Completioncited by 1
  31. 2022CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillationcited by 1
  32. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Frameworkcited by 1
  33. 2024Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentationcited by 1
  34. 2024Personalized Federated Learning for Generative AI-Assisted Semantic Communicationscited by 1
  35. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexitycited by 1
  36. 2025Dynamic Temperature Scheduler for Knowledge Distillationcited by 1
  37. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Modelscited by 1
  38. 2026Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learningcited by 1

50 most cited of 61 All 61 (JSON)