mlsanityData to 7 Oct 2026

Similarity-Preserving Knowledge Distillation

arXiv 1907.09682 ↗First posted 23 Jul 2019cs.CVcited by 507

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is a widely applicable technique for training a student neural network under the guidance of a trained teacher network. For example, in neural network compression, a high-capacity teacher is distilled to train a compact student; in privileged learning, a teacher trained with privileged data is distilled to train a student without access to that data. The distillation loss determines how a teacher's knowledge is captured and transferred to the student. In this paper, we propose a new form of knowledge distillation loss that is inspired by the observation that semantically similar inputs tend to elicit similar activation patterns in a trained network. Similarity-preserving knowledge distillation guides the training of a student network such that input pairs that produce similar (dissimilar) activations in the teacher network produce similar (dissimilar) activations in the student network. In contrast to previous distillation methods, the student is not required to mimic the representation space of the teacher, but rather to preserve the pairwise similarities in its own representation space. Experiments on three public datasets demonstrate the potential of our approach.

Key facts. Cited by 507 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 88; 2022: 131; 2023: 88; 2024: 96; 2025: 53; 2026: 51.

507 papers.

Sort:

By area:

  1. 2022Anomaly Detection via Reverse Distillation from One-Class Embeddingcited by 419
  2. 2022Decoupled Knowledge Distillationcited by 380
  3. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  4. 2023A Survey on Model Compression for Large Language Modelscited by 263
  5. 2022Knowledge Distillation from A Stronger Teachercited by 204
  6. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  7. 2023Getting aligned on representational alignmentcited by 142
  8. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  9. 2021Focal and Global Knowledge Distillation for Detectorscited by 138
  10. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 118
  11. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  12. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
38 more
  1. 2021General Instance Distillation for Object Detectioncited by 113
  2. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  3. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  4. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  5. 2024Logit Standardization in Knowledge Distillationcited by 101
  6. 2021Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognitioncited by 89
  7. 2023Deep Model Fusion: A Surveycited by 89
  8. 2022Unified Pretraining Framework for Document Understandingcited by 85
  9. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  10. 2023SCPNet: Semantic Scene Completion on Point Cloudcited by 80
  11. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  12. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  13. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  14. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  15. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
  16. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  17. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  18. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  19. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  20. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  21. 2021Lifelong Person Re-Identification via Adaptive Knowledge Accumulationcited by 57
  22. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  23. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  24. 2022LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detectioncited by 54
  25. 2021A Survey on Green Deep Learningcited by 52
  26. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  27. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  28. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  29. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  30. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  31. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  32. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  33. 2021Efficient Medical Image Segmentation Based on Knowledge Distillationcited by 43
  34. 2021Complementary Relation Contrastive Distillationcited by 42
  35. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  36. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficientcited by 41
  37. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  38. 2022NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Resultscited by 40

50 most cited of 507 All 507 (JSON)