mlsanityData to 14 Oct 2026

Cross-Layer Distillation with Semantic Calibration

arXiv 2012.03236 ↗First posted 6 Dec 2020cs.CV cs.AI cs.LGcited by 143

Not on the map. Why?

Cited by 96 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge distillation is a technique to enhance the generalization ability of a student model by exploiting outputs from a teacher model. Recently, feature-map based variants explore knowledge transfer between manually assigned teacher-student pairs in intermediate layers for further improvement. However, layer semantics may vary in different neural networks and semantic mismatch in manual layer associations will lead to performance degeneration due to negative regularization. To address this issue, we propose Semantic Calibration for cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple teacher layers rather than a specific intermediate layer for appropriate cross-layer supervision. We further provide theoretical analysis of the association weights and conduct extensive experiments to demonstrate the effectiveness of our approach. Code is avaliable at urlhttps://github.com/DefangChen/SemCKD.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 27; 2023: 38; 2024: 39; 2025: 19; 2026: 8.

143 papers.

Sort:

By area:

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  2. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
  3. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  4. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  5. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  6. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  7. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  8. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  9. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 40
  10. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 39
  11. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  12. 2022Linkless Link Prediction via Relational Distillationcited by 39
38 more
  1. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  2. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  3. 2023Visual Tuningcited by 34
  4. 2022Towards Efficient 3D Object Detection with Knowledge Distillationcited by 29
  5. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  6. 2022Improved Feature Distillation via Projector Ensemblecited by 27
  7. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
  8. 2021Distilling Holistic Knowledge with Graph Neural Networkscited by 26
  9. 2021Distilling a Powerful Student Model via Online Knowledge Distillationcited by 22
  10. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferabilitycited by 21
  11. 2023Function-Consistent Feature Distillationcited by 21
  12. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  13. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  14. 2023Coaching a Teachable Studentcited by 19
  15. 2023Distill n' Explain: explaining graph neural networks using simple surrogatescited by 18
  16. 2024SelfReg-UNet: Self-Regularized UNet for Medical Image Segmentationcited by 18
  17. 2023Adaptive Multi-Teacher Knowledge Distillation with Meta-Learningcited by 16
  18. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharingcited by 15
  19. 2022Accelerating Diffusion Sampling with Classifier-based Feature Distillationcited by 15
  20. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 14
  21. 2021Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distributioncited by 13
  22. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learningcited by 13
  23. 2023DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Drivingcited by 13
  24. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  25. 2024Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillationcited by 11
  26. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  27. 2021Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regressioncited by 10
  28. 2021On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networkscited by 9
  29. 2023Cross Architecture Distillation for Face Recognitioncited by 9
  30. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  31. 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Modelscited by 9
  32. 2021Semi-Online Knowledge Distillationcited by 8
  33. 2024Direct Distillation between Different Domainscited by 8
  34. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  35. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  36. 20223D Point Cloud Pre-training with Knowledge Distillation from 2D Imagescited by 7
  37. 2024NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Modelscited by 7
  38. 2023Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognitioncited by 6

50 most cited of 143 All 143 (JSON)