mlsanityData to 14 Oct 2026

CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation

arXiv 2504.21478 ↗First posted 30 Apr 2025cs.CV cs.NEcited by 2

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often neglecting the crucial aspect of the transferability of learned representations. In this paper, we propose Category-Aware Embedding Data-Free Knowledge Distillation (CAE-DFKD), which addresses at the embedding level the limitations of previous rely on image-level methods to improve model generalization but fail when directly applied to DFKD. The superiority and flexibility of CAE-DFKD are extensively evaluated, including: textiti.) Significant efficiency advantages resulting from altering the generator training paradigm; textitii.) Competitive performance with existing DFKD state-of-the-art methods on image recognition tasks; textitiii.) Remarkable transferability of data-free learned representations demonstrated in downstream tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 32 of its 41 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

32 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Small Scale Data-Free Knowledge Distillation12 Jun
  2. 2024De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts28 Mar
  3. 2023Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images20 Oct
  4. 2023Sampling to Distill: Knowledge Transfer from Open-World Data31 Jul
  5. 2023Class Attention Transfer Based Knowledge Distillation25 Apr
  6. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation28 Feb
  7. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation21 Sep
  8. 2022CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing24 May
  9. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay9 Jan
  10. 2021Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data11 Aug
  11. 2021Contrastive Model Inversion for Data-Free Knowledge Distillation18 May

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  5. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  6. 2019Data-Free Learning of Student Networkscited by 204
  7. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  8. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  9. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 80
  10. 2022Hard Sample Aware Network for Contrastive Deep Graph Clusteringcited by 61
  11. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 52
  12. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
9 more
  1. 2024RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Featurescited by 28
  2. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  3. 2021Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learningcited by 18
  4. 2022FakeCLR: Exploring Contrastive Learning for Solving Latent Discontinuity in Data-Efficient GANscited by 17
  5. 2022Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patchescited by 16
  6. 2022FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentationcited by 14
  7. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 13
  8. 2022ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Imagescited by 11
  9. 2022PointCLM: A Contrastive Learning-based Framework for Multi-instance Point Cloud Registrationcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  2. 2025Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Data-free Knowledge Distillation for Fine-grained Visual Categorization
  2. 2024Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation
  3. 2025Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation
  4. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability
  5. 2025FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer
  6. 2024Discriminative and Consistent Representation Distillation
  7. 2026UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
  8. 2026Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation