mlsanityData to 14 Oct 2026

Synthetic data generation method for data-free knowledge distillation in regression neural networks

arXiv 2301.04338 ↗First posted 11 Jan 2023cs.LGcited by 4

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger neural network as much as possible. Existing methods of knowledge distillation are mostly applicable for classification tasks. Many of them also require access to the data used to train the teacher model. To address the problem of knowledge distillation for regression tasks under the absence of original training data, previous work has proposed a data-free knowledge distillation method where synthetic data are generated using a generator model trained adversarially against the student model. These synthetic data and their labels predicted by the teacher model are then used to train the student model. In this study, we investigate the behavior of various synthetic data generation methods and propose a new synthetic data generation strategy that directly optimizes for a large but bounded difference between the student and teacher model. Our results on benchmark and case study experiments demonstrate that the proposed strategy allows the student model to learn better and emulate the performance of the teacher model more closely.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 23 of its 51 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

23 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Knowledge distillation with error-correcting transfer learning for wind power prediction1 Apr

Other references · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2021Convex optimizationcited by 2,515
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 464
  7. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  8. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  9. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  10. 2019Data-Free Learning of Student Networkscited by 204
  11. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  12. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
10 more
  1. 2019Deep Learning vs. Traditional Computer Visioncited by 94
  2. 2019Deep Learning for Audio Signal Processingcited by 92
  3. 2020Generative Adversarial Networks (GANs Survey): Challenges, Solutions, and Future Directionscited by 65
  4. 2018Learning Student Networks via Feature Embeddingcited by 45
  5. 2019Distilling Knowledge From a Deep Pose Regressor Networkcited by 34
  6. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  7. 2020Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashingcited by 26
  8. 2020Bringing AI To Edge: From Deep Learning's Perspectivecited by 24
  9. 2020Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillationcited by 18
  10. 2020An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
  2. 2026Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatchcited by 1
  3. 2026Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Neuromorphic Virtual Sensingcited by 1
  4. 2025SEEK: Self-adaptive Explainable Kernel For Nonstationary Gaussian Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Small Scale Data-Free Knowledge Distillation
  2. 2024Hybrid Data-Free Knowledge Distillation
  3. 2022Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay
  4. 2024Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation
  5. 2023Customizing Synthetic Data for Data-Free Student Learning
  6. 2021Conditional Generative Data-free Knowledge Distillation
  7. 2024Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks
  8. 2022Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation