mlsanityData to 14 Oct 2026

KD-DLGAN: Data Limited Image Generation via Knowledge Distillation

arXiv 2303.17158 ↗First posted 30 Mar 2023cs.CV eess.IVcited by 16

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Generative Adversarial Networks (GANs) rely heavily on large-scale training data for training high-quality image generation models. With limited training data, the GAN discriminator often suffers from severe overfitting which directly leads to degraded generation especially in generation diversity. Inspired by the recent advances in knowledge distillation (KD), we propose KD-DLGAN, a knowledge-distillation based generation framework that introduces pre-trained vision-language models for training effective data-limited generation models. KD-DLGAN consists of two innovative designs. The first is aggregated generative KD that mitigates the discriminator overfitting by challenging the discriminator with harder learning tasks and distilling more generalizable knowledge from the pre-trained models. The second is correlated generative KD that improves the generation diversity by distilling and preserving the diverse image-text correlation within the pre-trained models. Extensive experiments over multiple benchmarks show that KD-DLGAN achieves superior image generation with limited training data. In addition, KD-DLGAN complements the state-of-the-art with consistent and substantial performance gains.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 43 of its 57 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

43 of 57 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  10. 2017Improved Training of Wasserstein GANscited by 2,235
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
31 more
  1. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  2. 2016Least Squares Generative Adversarial Networkscited by 1,001
  3. 2018Self-Attention Generative Adversarial Networkscited by 809
  4. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  5. 2019Similarity-Preserving Knowledge Distillationcited by 516
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  8. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  9. 2018cGANs with Projection Discriminatorcited by 254
  10. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  11. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  12. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  13. 2015Cross Modal Distillation for Supervision Transfercited by 175
  14. 2019EdgeConnect: Generative Image Inpainting with Adversarial Edge Learningcited by 163
  15. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  16. 2020Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANscited by 132
  17. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 118
  18. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 107
  19. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 86
  20. 2022Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillationcited by 68
  21. 2021Data-Efficient Instance Generation from Instance Discriminationcited by 58
  22. 2020Diverse Image Generation via Self-Conditioned GANscited by 55
  23. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  24. 2021Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Datacited by 53
  25. 2020Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillationcited by 47
  26. 2019Detecting Overfitting of Deep Generative Networks via Latent Recoverycited by 44
  27. 2021Unbalanced Feature Transport for Exemplar-based Image Translationcited by 41
  28. 2020Towards GAN Benchmarks Which Require Generalizationcited by 31
  29. 2021Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillationcited by 22
  30. 2022Towards Counterfactual Image Manipulation via CLIPcited by 19
  31. 2021GenCo: Generative Co-training for Generative Adversarial Networks with Limited Datacited by 14

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 6; 2025: 6.

16 papers.

Sort:

By area:

  1. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  2. 2023A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shotcited by 19
  3. 2024Weakly Supervised Monocular 3D Detection with a Single-View Imagecited by 10
  4. 2023Image Synthesis under Limited Data: A Survey and Taxonomycited by 9
  5. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  6. 2025VRM: Knowledge Distillation via Virtual Relation Matchingcited by 6
  7. 2025Cross-Architecture Distillation Made Simple with Redundancy Suppressioncited by 4
  8. 2024HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classificationcited by 3
  9. 2023RetriBooru: Leakage-Free Retrieval of Conditions from Reference Images for Subject-Driven Generationcited by 2
  10. 2024CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalizationcited by 2
  11. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Imagescited by 2
  12. 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
4 more
  1. 2023Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and Uncurated Unlabeled Datacited by 0
  2. 2024Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictionscited by 0
  3. 2025Style Quantization for Data-Efficient GAN Trainingcited by 0
  4. 2025Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Diversity in Black-box Few-shot Knowledge Distillation
  2. 2025SDAKD: Student Discriminator Assisted Knowledge Distillation for Super-Resolution Generative Adversarial Networks
  3. 2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation
  4. 2022Discriminator-Cooperated Feature Map Distillation for GAN Compression
  5. 2021Conditional Generative Data-free Knowledge Distillation
  6. 2021Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression
  7. 2022Region-aware Knowledge Distillation for Efficient Image-to-Image Translation
  8. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed