Data to 14 Oct 2026

Contrastive Representation Distillation

arXiv 1910.10699 ↗First posted 23 Oct 2019cs.LG cs.CV stat.MLcited by 748

Not on the map. Why?

Cited by 402 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation. Code: http://github.com/HobbitLong/RepDistiller.

Key facts. Cited by 748 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 129; 2022: 156; 2023: 148; 2024: 124; 2025: 76; 2026: 115.

748 papers.

Sort:

By area:

2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 436
2022Decoupled Knowledge Distillationcited by 388
2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 348
2021Distilling Knowledge via Knowledge Reviewcited by 288
2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
2022Knowledge Distillation from A Stronger Teachercited by 208
2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
2023Getting aligned on representational alignmentcited by 143
2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
2021SEED: Self-supervised Distillation For Visual Representationcited by 134

38 more

2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
2023Dataset Distillation: A Comprehensive Reviewcited by 127
2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
2022Asymmetric Student-Teacher Networks for Industrial Anomaly Detectioncited by 117
2021General Instance Distillation for Object Detectioncited by 114
2022Curriculum Temperature for Knowledge Distillationcited by 112
2022Masked Generative Distillationcited by 107
2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
2024Logit Standardization in Knowledge Distillationcited by 102
2023Diffusion Model as Representation Learnercited by 95
2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 79
2021Vision Transformer Pruningcited by 79
2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 78
2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
2022Knowledge Distillation via the Target-aware Transformercited by 70
2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
2022ALSO: Automotive Lidar Self-supervision by Occupancy estimationcited by 68
2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 64
2021Graph Contrastive Clusteringcited by 61
2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
2023Class Attention Transfer Based Knowledge Distillationcited by 61
2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 60
2021Fair Feature Distillation for Visual Recognitioncited by 59
2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
2022MAViL: Masked Audio-Video Learnerscited by 59
2021Initialization and Regularization of Factorized Neural Layerscited by 58
2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
2023Decoupled Kullback-Leibler Divergence Losscited by 58
2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57

50 most cited of 748 All 748 (JSON)