mlsanityData to 14 Oct 2026

Information Theoretic Representation Distillation

arXiv 2112.00459 ↗First posted 1 Dec 2021cs.CVcited by 19

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Despite the empirical success of knowledge distillation, current state-of-the-art methods are computationally expensive to train, which makes them difficult to adopt in practice. To address this problem, we introduce two distinct complementary losses inspired by a cheap entropy-like estimator. These losses aim to maximise the correlation and mutual information between the student and teacher representations. Our method incurs significantly less training overheads than other approaches and achieves competitive performance to the state-of-the-art on the knowledge distillation and cross-model transfer tasks. We further demonstrate the effectiveness of our method on a binary distillation task, whereby it leads to a new state-of-the-art for binary quantisation and approaches the performance of a full precision model. Code: www.github.com/roymiles/ITRD

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 51 of its 69 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

51 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Hierarchical Self-supervised Augmented Knowledge Distillation29 Jul
  2. 2021Distilling Knowledge via Knowledge Review19 Apr
  3. 2021Complementary Relation Contrastive Distillation29 Mar

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2016Wide Residual Networkscited by 3,070
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,653
36 more
  1. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  5. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  6. 2019Relational Knowledge Distillationcited by 889
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  9. 2019Contrastive Representation Distillationcited by 748
  10. 2017Renyi Differential Privacycited by 650
  11. 2019Similarity-Preserving Knowledge Distillationcited by 516
  12. 2019Variational Information Distillation for Knowledge Transfercited by 324
  13. 2018Slimmable Neural Networkscited by 305
  14. 2019Correlation Congruence for Knowledge Distillationcited by 285
  15. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  16. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  17. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  18. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  19. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  20. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  21. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  22. 2016FINN: A Framework for Fast, Scalable Binarized Neural Network Inferencecited by 150
  23. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  24. 2020Rotated Binary Neural Networkcited by 77
  25. 2020Wasserstein Contrastive Representation Distillationcited by 64
  26. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  27. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 52
  28. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 32
  29. 2020MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?cited by 32
  30. 2018Understanding Autoencoders with Information Theoretic Conceptscited by 29
  31. 2020Prime-Aware Adaptive Distillationcited by 28
  32. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 26
  33. 2020torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillationcited by 19
  34. 2021Deep Deterministic Information Bottleneck with Matrix-based Entropy Functionalcited by 18
  35. 2013Information Theoretic Learning with Infinitely Divisible Kernelscited by 9
  36. 2020Cascaded channel pruning using hierarchical self-distillationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 3; 2024: 6; 2025: 5; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  2. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  3. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  4. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  5. 2023MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillationcited by 16
  6. 2024Quantifying Knowledge Distillation Using Partial Information Decompositioncited by 13
  7. 2023Multi-Label Knowledge Distillationcited by 12
  8. 2022Optimal Randomized Approximations for Matrix based Renyi's Entropycited by 7
  9. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learningcited by 5
  10. 2025DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detectioncited by 5
  11. 2021Computationally Efficient Approximations for Matrix-based Renyi's Entropycited by 4
  12. 2025HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentationcited by 3
7 more
  1. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfercited by 3
  2. 2022Robust and Fast Measure of Information via Low-rank Representationcited by 1
  3. 2025Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgettingcited by 1
  4. 2024Learning to Project for Cross-Task Knowledge Distillationcited by 0
  5. 2024On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Modelscited by 0
  6. 2024Discriminative and Consistent Representation Distillationcited by 0
  7. 2026LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Rethinking Knowledge Distillation via Cross-Entropy
  2. 2021Knowledge Distillation as Semiparametric Inference
  3. 2024Relational Representation Distillation
  4. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation
  5. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic View
  6. 2024How to Train the Teacher Model for Effective Knowledge Distillation
  7. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
  8. 2024Linear Projections of Teacher Embeddings for Few-Class Distillation