mlsanityData to 7 Oct 2026

Relational Knowledge Distillation

arXiv 1904.05068 ↗First posted 10 Apr 2019cs.CV cs.LGcited by 873

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.

Key facts. Cited by 873 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 115; 2022: 139; 2023: 171; 2024: 172; 2025: 132; 2026: 144.

873 papers.

Sort:

By area:

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2022Decoupled Knowledge Distillationcited by 380
  5. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  6. 2023A Survey on Model Compression for Large Language Modelscited by 263
  7. 2023Class-Incremental Learning: A Surveycited by 209
  8. 2022Knowledge Distillation from A Stronger Teachercited by 204
  9. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  10. 2023Getting aligned on representational alignmentcited by 142
  11. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  12. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
38 more
  1. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  2. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  3. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  4. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 118
  5. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  6. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  7. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  8. 2021General Instance Distillation for Object Detectioncited by 113
  9. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  10. 2022Masked Generative Distillationcited by 107
  11. 2024Logit Standardization in Knowledge Distillationcited by 101
  12. 2023Diffusion Model as Representation Learnercited by 95
  13. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  14. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  15. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  16. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  17. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  18. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  19. 2024NetLLM: Adapting Large Language Models for Networkingcited by 76
  20. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  21. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  22. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  23. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
  24. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  25. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  26. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  27. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  28. 2021Fair Feature Distillation for Visual Recognitioncited by 59
  29. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  30. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  31. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  32. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  33. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  34. 2022MAViL: Masked Audio-Video Learnerscited by 56
  35. 2021Localization Distillation for Dense Object Detectioncited by 55
  36. 2022Multimodal learning with graphscited by 54
  37. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  38. 2023Knowledge Diffusion for Distillationcited by 51

50 most cited of 873 All 873 (JSON)