mlsanityData to 14 Oct 2026

Correlation Congruence for Knowledge Distillation

arXiv 1904.01802 ↗First posted 3 Apr 2019cs.CVcited by 285

Not on the map. Why?

Cited by 188 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information, but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and achieves state-of-the-art accuracy compared with other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hint-based learning methods.

Key facts. Cited by 285 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 49; 2022: 75; 2023: 52; 2024: 56; 2025: 32; 2026: 21.

285 papers.

Sort:

By area:

  1. 2022Decoupled Knowledge Distillationcited by 388
  2. 2022Knowledge Distillation from A Stronger Teachercited by 208
  3. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  4. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 118
  5. 2022Curriculum Temperature for Knowledge Distillationcited by 112
  6. 2024Logit Standardization in Knowledge Distillationcited by 102
  7. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 79
  8. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 78
  9. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  10. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
  11. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  12. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
38 more
  1. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillationcited by 60
  2. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  3. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  4. 2021A Survey on Green Deep Learningcited by 52
  5. 2021Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimationcited by 48
  6. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  7. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  8. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  9. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  10. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  11. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  12. 2022Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classificationcited by 43
  13. 2021Complementary Relation Contrastive Distillationcited by 42
  14. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  15. 2022Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Taskscited by 42
  16. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  17. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  18. 2021CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identificationcited by 38
  19. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  20. 2021Learning Compatible Embeddingscited by 37
  21. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  22. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  23. 2023Visual Tuningcited by 34
  24. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 33
  25. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  26. 2023TinySAM: Pushing the Envelope for Efficient Segment Anything Modelcited by 33
  27. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  28. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  29. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  30. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  31. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 27
  32. 2022Improved Feature Distillation via Projector Ensemblecited by 27
  33. 2021Distilling Holistic Knowledge with Graph Neural Networkscited by 26
  34. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  35. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  36. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  37. 2024DVMSR: Distillated Vision Mamba for Efficient Super-Resolutioncited by 24
  38. 2022Proto2Proto: Can you recognize the car, the way I do?cited by 23

50 most cited of 285 All 285 (JSON)