# UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

arXiv: https://arxiv.org/abs/2608.09287 · HTML: https://mlsanity.com/paper/2608.09287/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Modulating early visual processing by language](https://mlsanity.com/paper/1707.00683/) (2017)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Patch Similarity Aware Data-Free Quantization for Vision Transformers](https://mlsanity.com/paper/2203.02250/) (2022)
- [A Comprehensive Survey on Knowledge Distillation](https://mlsanity.com/paper/2503.12067/) (2025)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2209.10359/) (2022)
- [NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation](https://mlsanity.com/paper/2310.00258/) (2023)
- [Small Scale Data-Free Knowledge Distillation](https://mlsanity.com/paper/2406.07876/) (2024)
- [CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs](https://mlsanity.com/paper/2407.05266/) (2024)
- [Data-free Knowledge Distillation with Diffusion Models](https://mlsanity.com/paper/2504.00870/) (2025)

## Abstract (arXiv)

Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset. Existing DFKD methods rely heavily on architecture-specific statistical priors (e.g., Batch Normalization statistics) to guide data synthesis, however, such architecture-dependent priors are often absent in modern architectures such as Vision Transformers (ViTs), resulting in degraded semantic quality of the synthesized data and consequently catastrophic performance degradation. In this paper, we propose UniDFKD, a unified data-free knowledge distillation framework that replaces architecture-specific statistics with explicit, architecture-agnostic semantic priors. UniDFKD governs the entire synthesis-distillation pipeline along three dimensions: (1) Categorical Semantic Conditioning (CSC) defines what to synthesize by persistently modulating the generator with language-derived embeddings to capture semantic diversity; (2) Spatial Semantic Anchoring (SSA) dictates where evidence belongs by anchoring the teacher's spatial attributions to a Gaussian prior; and (3) Spatial Semantic Distillation (SSD) controls how knowledge is transferred by explicitly aligning teacher-student spatial evidence alongside predictions. Extensive experiments across CNNs and ViTs demonstrate that UniDFKD establishes a new state-of-the-art, outperforming existing methods by an average absolute margin of over 20% in both homogeneous and heterogeneous settings.