mlsanityData to 14 Oct 2026

Improved knowledge distillation by utilizing backward pass knowledge in neural networks

arXiv 2301.12006 ↗First posted 27 Jan 2023cs.LG cs.CL cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation (KD) is one of the prominent techniques for model compression. In this method, the knowledge of a large network (teacher) is distilled into a model (student) with usually significantly fewer parameters. KD tries to better-match the output of the student model to that of the teacher model based on the knowledge extracts from the forward pass of the teacher network. Although conventional KD is effective for matching the two networks over the given data points, there is no guarantee that these models would match in other areas for which we do not have enough training samples. In this work, we address that problem by generating new auxiliary training samples based on extracting knowledge from the backward pass of the teacher in the areas where the student diverges greatly from the teacher. We compute the difference between the teacher and the student and generate new data samples that maximize the divergence. This is done by perturbing data samples in the direction of the gradient of the difference between the student and the teacher. Augmenting the training set by adding this auxiliary improves the performance of KD significantly and leads to a closer match between the student and the teacher. Using this approach, when data samples come from a discrete domain, such as applications of natural language processing (NLP) and language understanding, is not trivial. However, we show how this technique can be used successfully in such applications. We evaluated the performance of our method on various tasks in computer vision and NLP domains and got promising results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 22 of its 29 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

22 of 29 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
10 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  3. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  4. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  5. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
  6. 2018Knowledge Transfer with Jacobian Matchingcited by 76
  7. 2019Pruning from Scratchcited by 72
  8. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  9. 2019YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detectioncited by 16
  10. 2018Tensor Decomposition for Compressing Recurrent Neural Networkcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022A Closer Look at Knowledge Distillation with Features, Logits, and Gradients
  2. 2024Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching
  3. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
  4. 2024Improve Knowledge Distillation via Label Revision and Data Selection
  5. 2023What is Lost in Knowledge Distillation?
  6. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
  7. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
  8. 2021Annealing Knowledge Distillation