mlsanityData to 14 Oct 2026

KD2M: A unifying framework for feature knowledge distillation

arXiv 2504.01757 ↗First posted 2 Apr 2025stat.ML cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge Distillation (KD) seeks to transfer the knowledge of a teacher, towards a student neural net. This process is often done by matching the networks' predictions (i.e., their output), but, recently several works have proposed to match the distributions of neural nets' activations (i.e., their features), a process known as distribution matching. In this paper, we propose an unifying framework, Knowledge Distillation through Distribution Matching (KD2M), which formalizes this strategy. Our contributions are threefold. We i) provide an overview of distribution metrics used in distribution matching, ii) benchmark on computer vision datasets, and iii) derive new theoretical results for KD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 16 of its 24 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

16 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation11 Dec

Other references · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2021Dataset Condensation with Distribution Matchingcited by 365
  6. 2017Joint Distribution Optimal Transportation for Domain Adaptationcited by 282
  7. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  8. 2016Theoretical Analysis of Domain Adaptation with Optimal Transportcited by 105
  9. 2018An elementary introduction to information geometrycited by 76
  10. 2023Recent Advances in Optimal Transport for Machine Learningcited by 66
  11. 2020Wasserstein Contrastive Representation Distillationcited by 64
  12. 2023Dataset Distillation via the Wasserstein Metriccited by 31
3 more
  1. 2023Multi-Source Domain Adaptation through Dataset Dictionary Learning in Wasserstein Spacecited by 12
  2. 2020Model Compression Using Optimal Transportcited by 8
  3. 2023Multi-Source Domain Adaptation meets Dataset Distillation through Dataset Dictionary Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability
  2. 2024A Comprehensive Review of Knowledge Distillation in Computer Vision
  3. 2024Harmonizing knowledge Transfer in Neural Network with Unified Distillation
  4. 2022Towards a Unified View of Affinity-Based Knowledge Distillation
  5. 2023Distribution Shift Matters for Knowledge Distillation with Webly Collected Images
  6. 2025Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation
  7. 2022Knowledge Distillation from A Stronger Teacher
  8. 2021Natural Statistics of Network Activations and Implications for Knowledge Distillation