mlsanityData to 14 Oct 2026

Variational Information Distillation for Knowledge Transfer

arXiv 1904.05835 ↗First posted 11 Apr 2019cs.CV cs.AI cs.LGcited by 324

Not on the map. Why?

Cited by 197 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.

Key facts. Cited by 324 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 65; 2022: 78; 2023: 56; 2024: 63; 2025: 30; 2026: 32.

324 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  2. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 324
  3. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  4. 2022Knowledge Distillation from A Stronger Teachercited by 208
  5. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  6. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
  7. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  8. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  9. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  10. 2021General Instance Distillation for Object Detectioncited by 114
  11. 2022Curriculum Temperature for Knowledge Distillationcited by 112
  12. 2021Distilling Object Detectors via Decoupled Featurescited by 106
38 more
  1. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  2. 2024Logit Standardization in Knowledge Distillationcited by 102
  3. 2023Deep Model Fusion: A Surveycited by 91
  4. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  5. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 79
  6. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  7. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 60
  8. 2021Fair Feature Distillation for Visual Recognitioncited by 59
  9. 2023Knowledge Diffusion for Distillationcited by 53
  10. 2021A Survey on Green Deep Learningcited by 52
  11. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  12. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  13. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  14. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  15. 2021ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalitiescited by 43
  16. 2021Complementary Relation Contrastive Distillationcited by 42
  17. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  18. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficientcited by 41
  19. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  20. 2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39
  21. 2022Novel Class Discovery without Forgettingcited by 39
  22. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  23. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  24. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  25. 2021AlphaNet: Improved Training of Supernets with Alpha-Divergencecited by 36
  26. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  27. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 33
  28. 2022Locality Guidance for Improving Vision Transformers on Tiny Datasetscited by 33
  29. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  30. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  31. 2021The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactionscited by 31
  32. 2021Dual-Teacher Class-Incremental Learning With Data-Free Generative Replaycited by 31
  33. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  34. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  35. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  36. 2022Knowledge Distillation based Degradation Estimation for Blind Super-Resolutioncited by 28
  37. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  38. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 27

50 most cited of 324 All 324 (JSON)