mlsanityData to 7 Oct 2026

Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis

arXiv 2104.04868 ↗First posted 10 Apr 2021cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) has proved to be an effective approach for deep neural network compression, which learns a compact network (student) by transferring the knowledge from a pre-trained, over-parameterized network (teacher). In traditional KD, the transferred knowledge is usually obtained by feeding training samples to the teacher network to obtain the class probabilities. However, the original training dataset is not always available due to storage costs or privacy issues. In this study, we propose a novel data-free KD approach by modeling the intermediate feature space of the teacher with a multivariate normal distribution and leveraging the soft targeted labels generated by the distribution to synthesize pseudo samples as the transfer set. Several student networks trained with these synthesized transfer sets present competitive performance compared to the networks trained with the original training set and other data-free KD approaches.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 10 Apr 2021.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2017Improved Training of Wasserstein GANscited by 2,223
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2014How transferable are features in deep neural networks?cited by 1,216
19 more
  1. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2013Do Deep Nets Really Need to be Deep?cited by 466
  5. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  6. 2019Variational Information Distillation for Knowledge Transfercited by 321
  7. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  8. 2019Data-Free Learning of Student Networkscited by 203
  9. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  10. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  11. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  12. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 119
  13. 2019Pruning from Scratchcited by 72
  14. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  15. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  16. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  17. 2020Uncertainty-Aware Multi-Shot Knowledge Distillation for Image-Based Object Re-Identificationcited by 31
  18. 2018Few-shot learning of neural networks from scratch by pseudo example optimizationcited by 15
  19. 2018Fast-converging Conditional Generative Adversarial Networks for Image Synthesiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 4; 2025: 1; 2026: 1.

14 papers.

Sort:

  1. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  2. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillationcited by 31
  3. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
  4. 2022Black-box Few-shot Knowledge Distillationcited by 14
  5. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  6. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 8
  7. 2022Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategycited by 7
  8. 2023Deep Classifier Mimicry without Data Accesscited by 6
  9. 2023Task-Distributionally Robust Data-Free Meta-Learningcited by 4
  10. 2025Tractable Representation Learning with Probabilistic Circuitscited by 4
  11. 2023Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Imagescited by 2
  12. 2024Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generatorcited by 2
2 more
  1. 2024DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learningcited by 2
  2. 2026Improving Diversity in Black-box Few-shot Knowledge Distillationcited by 0