mlsanityData to 7 Oct 2026

Knowledge Condensation Distillation

arXiv 2207.05409 ↗First posted 12 Jul 2022cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) transfers the knowledge from a high-capacity teacher network to strengthen a smaller student. Existing methods focus on excavating the knowledge hints and transferring the whole knowledge to the student. However, the knowledge redundancy arises since the knowledge shows different values to the student at different learning stages. In this paper, we propose Knowledge Condensation Distillation (KCD). Specifically, the knowledge value on each sample is dynamically estimated, based on which an Expectation-Maximization (EM) framework is forged to iteratively condense a compact knowledge set from the teacher to guide the student learning. Our approach is easy to build on top of the off-the-shelf KD methods, with no extra training parameters and negligible computation overhead. Thus, it presents one new perspective for KD, in which the student that actively identifies teacher's knowledge in line with its aptitude can learn to learn more effectively and efficiently. Experiments on standard benchmarks manifest that the proposed KCD can well boost the performance of student model with even higher distillation efficiency. Code is available at https://github.com/dzy3/KCD.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

36 of 43 references.

References · 36

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2016Wide Residual Networkscited by 3,057
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  12. 2019When Does Label Smoothing Help?cited by 985
24 more
  1. 2019Relational Knowledge Distillationcited by 873
  2. 2019Contrastive Representation Distillationcited by 734
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  4. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  5. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  6. 2020Dataset Condensation with Gradient Matchingcited by 510
  7. 2019Similarity-Preserving Knowledge Distillationcited by 507
  8. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  9. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  11. 2019Variational Information Distillation for Knowledge Transfercited by 321
  12. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  13. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  14. 2018Dataset Distillationcited by 224
  15. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  16. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  17. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  18. 2020Wasserstein Contrastive Representation Distillationcited by 63
  19. 2021Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Studycited by 50
  20. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  21. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 40
  22. 2020Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixupcited by 22
  23. 2021Distilling a Powerful Student Model via Online Knowledge Distillationcited by 21
  24. 2020Learning Efficient GANs for Image Translation via Differentiable Masks and co-Attention Distillationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 12; 2025: 2; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  2. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  3. 2024Endora: Video Generation Models as Endoscopy Simulatorscited by 53
  4. 2022StegaNeRF: Embedding Invisible Information within Neural Radiance Fieldscited by 32
  5. 2024LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstructioncited by 21
  6. 2024EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splattingcited by 16
  7. 2024DiffRect: Latent Diffusion Label Rectification for Semi-supervised Medical Image Segmentationcited by 14
  8. 2024GaussianStego: A Generalizable Stenography Pipeline for Generative 3D Gaussians Splattingcited by 11
  9. 2024De-LightSAM: Modality-Decoupled Lightweight SAM for Generalizable Medical Segmentationcited by 11
  10. 2024Improve Cross-Architecture Generalization on Dataset Distillationcited by 7
  11. 2024GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representationcited by 7
  12. 2022Discriminator-Cooperated Feature Map Distillation for GAN Compressioncited by 4
8 more
  1. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
  2. 2024Instance Temperature Knowledge Distillationcited by 3
  3. 2025ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian Splattingcited by 3
  4. 2025Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Studentcited by 2
  5. 2026Beyond Sparse Supervision: Diffusion-Guided Learning for Few-Shot Graph Fraud Detectioncited by 2
  6. 2024BACON: Bayesian Optimal Condensation Framework for Dataset Distillationcited by 1
  7. 2022Hint-dynamic Knowledge Distillationcited by 0
  8. 2024ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNscited by 0