mlsanityData to 14 Oct 2026

Learning Deep Representations with Probabilistic Knowledge Transfer

arXiv 1803.10837 ↗First posted 28 Mar 2018cs.LG cs.NE stat.MLcited by 213

Not on the map. Why?

Cited by 133 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge Transfer (KT) techniques tackle the problem of transferring the knowledge from a large and complex neural network into a smaller and faster one. However, existing KT methods are tailored towards classification tasks and they cannot be used efficiently for other representation learning tasks. In this paper a novel knowledge transfer technique, that is capable of training a student model that maintains the same amount of mutual information between the learned representation and a set of (possible unknown) labels as the teacher model, is proposed. Apart from outperforming existing KT techniques, the proposed method allows for overcoming several limitations of existing methods providing new insight into KT as well as novel KT applications, ranging from knowledge transfer from handcrafted feature extractors to {cross-modal} KT from the textual modality into the representation extracted from the visual modality of the data.

Key facts. Cited by 213 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 45; 2023: 49; 2024: 56; 2025: 21; 2026: 10.

213 papers.

Sort:

By area:

  1. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  2. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
  3. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  4. 2022Curriculum Temperature for Knowledge Distillationcited by 112
  5. 2021Distilling Object Detectors via Decoupled Featurescited by 106
  6. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  7. 2023SCPNet: Semantic Scene Completion on Point Cloudcited by 80
  8. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  9. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
  10. 2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63
  11. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  12. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 60
38 more
  1. 2021Fair Feature Distillation for Visual Recognitioncited by 59
  2. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  3. 2022Factorizing Knowledge in Neural Networkscited by 59
  4. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  5. 2021ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Imagescited by 48
  6. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  7. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  8. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  9. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  10. 2021Complementary Relation Contrastive Distillationcited by 42
  11. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  12. 2023MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learningcited by 40
  13. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  14. 2023A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversationscited by 37
  15. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  16. 2021AlphaNet: Improved Training of Supernets with Alpha-Divergencecited by 36
  17. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  18. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  19. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  20. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  21. 2021Distilling Holistic Knowledge with Graph Neural Networkscited by 26
  22. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  23. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  24. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  25. 2024Frequency Attention for Knowledge Distillationcited by 24
  26. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  27. 2022Spot-adaptive Knowledge Distillationcited by 22
  28. 2021Embedding Transfer with Label Relaxation for Improved Metric Learningcited by 21
  29. 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillationcited by 21
  30. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferabilitycited by 21
  31. 2024Direct Preference Knowledge Distillation for Large Language Modelscited by 21
  32. 2022Knowledge Condensation Distillationcited by 20
  33. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  34. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  35. 2021Information Theoretic Representation Distillationcited by 19
  36. 2021Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teachingcited by 19
  37. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  38. 2023Coaching a Teachable Studentcited by 19

50 most cited of 213 All 213 (JSON)