# CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation

arXiv: https://arxiv.org/abs/2504.21478 · HTML: https://mlsanity.com/paper/2504.21478/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Hard Sample Aware Network for Contrastive Deep Graph Clustering](https://mlsanity.com/paper/2212.08665/) (2022)
- [Class Attention Transfer Based Knowledge Distillation](https://mlsanity.com/paper/2304.12777/) (2023)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [Delving into Effective Gradient Matching for Dataset Condensation](https://mlsanity.com/paper/2208.00311/) (2022)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)
- [Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data](https://mlsanity.com/paper/2108.05698/) (2021)
- [Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2209.10359/) (2022)
- [Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation](https://mlsanity.com/paper/2302.14290/) (2023)
- [RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features](https://mlsanity.com/paper/2403.05061/) (2024)
- [NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation](https://mlsanity.com/paper/2310.00258/) (2023)
- [Small Scale Data-Free Knowledge Distillation](https://mlsanity.com/paper/2406.07876/) (2024)
- [Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning](https://mlsanity.com/paper/2111.14585/) (2021)
- [FakeCLR: Exploring Contrastive Learning for Solving Latent Discontinuity in Data-Efficient GANs](https://mlsanity.com/paper/2207.08630/) (2022)
- [Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches](https://mlsanity.com/paper/2207.08220/) (2022)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2208.14143/) (2022)
- [Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation](https://mlsanity.com/paper/2410.17606/) (2024)
- [ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Images](https://mlsanity.com/paper/2207.06733/) (2022)
- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [Sampling to Distill: Knowledge Transfer from Open-World Data](https://mlsanity.com/paper/2307.16601/) (2023)

## Built on by (most cited first, 2 of 2)

- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimization](https://mlsanity.com/paper/2506.06992/) (2025)

## Abstract (arXiv)

Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often neglecting the crucial aspect of the transferability of learned representations. In this paper, we propose Category-Aware Embedding Data-Free Knowledge Distillation (CAE-DFKD), which addresses at the embedding level the limitations of previous rely on image-level methods to improve model generalization but fail when directly applied to DFKD. The superiority and flexibility of CAE-DFKD are extensively evaluated, including: textiti.) Significant efficiency advantages resulting from altering the generator training paradigm; textitii.) Competitive performance with existing DFKD state-of-the-art methods on image recognition tasks; textitiii.) Remarkable transferability of data-free learned representations demonstrated in downstream tasks.