mlsanityData to 7 Oct 2026

Complementary Relation Contrastive Distillation

arXiv 2103.16367 ↗First posted 29 Mar 2021cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation aims to transfer representation ability from a teacher model to a student model. Previous approaches focus on either individual representation distillation or inter-sample similarity preservation. While we argue that the inter-sample relation conveys abundant information and needs to be distilled in a more effective way. In this paper, we propose a novel knowledge distillation method, namely Complementary Relation Contrastive Distillation (CRCD), to transfer the structural knowledge from the teacher to the student. Specifically, we estimate the mutual relation in an anchor-based way and distill the anchor-student relation under the supervision of its corresponding anchor-teacher relation. To make it more robust, mutual relations are modeled by two complementary elements: the feature and its gradient. Furthermore, the low bound of mutual information between the anchor-teacher relation distribution and the anchor-student relation distribution is maximized via relation contrastive loss, which can distill both the sample representation and the inter-sample relations. Experiments on different benchmarks demonstrate the effectiveness of our proposed CRCD.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

37 of 52 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2016Wide Residual Networkscited by 3,057
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
25 more
  1. 2020Knowledge Distillation: A Surveycited by 1,578
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2019Relational Knowledge Distillationcited by 873
  9. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  10. 2019Contrastive Representation Distillationcited by 734
  11. 2019Similarity-Preserving Knowledge Distillationcited by 507
  12. 2019Variational Information Distillation for Knowledge Transfercited by 321
  13. 2019Correlation Congruence for Knowledge Distillationcited by 280
  14. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  15. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  16. 2020Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identificationcited by 214
  17. 2020Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-IDcited by 212
  18. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  19. 2017Image-Image Domain Adaptation with Preserved Self-Similarity and Domain-Dissimilarity for Person Re-identificationcited by 168
  20. 2019Ensemble Distribution Distillationcited by 158
  21. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  22. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  23. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  24. 2019Memory-Based Neighbourhood Embedding for Visual Recognitioncited by 16
  25. 2018Adversarial Network Compressioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 12; 2023: 8; 2024: 12; 2025: 3; 2026: 3.

42 papers.

Sort:

By area:

  1. 2022Anomaly Detection via Reverse Distillation from One-Class Embeddingcited by 419
  2. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  3. 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51
  4. 2022Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Taskscited by 42
  5. 2023Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learningcited by 34
  6. 2022Switchable Online Knowledge Distillationcited by 29
  7. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  8. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  9. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  10. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  11. 2021Information Theoretic Representation Distillationcited by 19
  12. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
30 more
  1. 2022Generalized Knowledge Distillation via Relationship Matchingcited by 12
  2. 2022Distilling Knowledge from Object Classification to Aesthetics Assessmentcited by 12
  3. 2023Enhanced Multimodal Representation Learning with Cross-modal KDcited by 11
  4. 2024Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillationcited by 11
  5. 2022Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domainscited by 10
  6. 2022Region-aware Knowledge Distillation for Efficient Image-to-Image Translationcited by 10
  7. 2023Distribution Shift Matters for Knowledge Distillation with Webly Collected Imagescited by 8
  8. 2023Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentationcited by 8
  9. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 6
  10. 2024Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Biascited by 5
  11. 2024Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillationcited by 3
  12. 2025Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Modelscited by 3
  13. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  14. 2021Optimizing for In-memory Deep Learning with Emerging Memory Technologycited by 1
  15. 2021Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognitioncited by 1
  16. 2022Seeing your sleep stage: cross-modal distillation from EEG to infrared videocited by 1
  17. 2022Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolutioncited by 1
  18. 2023Topology-preserving Adversarial Training for Alleviating Natural Accuracy Degradationcited by 1
  19. 2024Collaborative Knowledge Distillation via a Learning-by-Education Node Communitycited by 1
  20. 2022Towards a Unified View of Affinity-Based Knowledge Distillationcited by 0
  21. 2022Hint-dynamic Knowledge Distillationcited by 0
  22. 2024Discriminative and Consistent Representation Distillationcited by 0
  23. 2024PRG: Prompt-Based Distillation Without Annotation via Proxy Relational Graphcited by 0
  24. 2024Low-Resolution Face Recognition via Adaptable Instance-Relation Distillationcited by 0
  25. 2024Preview-based Category Contrastive Learning for Knowledge Distillationcited by 0
  26. 2025Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspectivecited by 0
  27. 2025SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learningcited by 0
  28. 2026Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Modelcited by 0
  29. 2026Training a Student Expert via Semi-Supervised Foundation Model Distillationcited by 0
  30. 2026TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognitioncited by 0