# KD-DLGAN: Data Limited Image Generation via Knowledge Distillation

arXiv: https://arxiv.org/abs/2303.17158 · HTML: https://mlsanity.com/paper/2303.17158/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs](https://mlsanity.com/paper/2002.10964/) (2020)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)

## Built on by (most cited first, 16 of 16)

- [VkD: Improving Knowledge Distillation using Orthogonal Projections](https://mlsanity.com/paper/2403.06213/) (2024)
- [A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shot](https://mlsanity.com/paper/2307.14397/) (2023)
- [Weakly Supervised Monocular 3D Detection with a Single-View Image](https://mlsanity.com/paper/2402.19144/) (2024)
- [Image Synthesis under Limited Data: A Survey and Taxonomy](https://mlsanity.com/paper/2307.16879/) (2023)
- [CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective](https://mlsanity.com/paper/2404.14109/) (2024)
- [VRM: Knowledge Distillation via Virtual Relation Matching](https://mlsanity.com/paper/2502.20760/) (2025)
- [Cross-Architecture Distillation Made Simple with Redundancy Suppression](https://mlsanity.com/paper/2507.21844/) (2025)
- [HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification](https://mlsanity.com/paper/2407.07516/) (2024)
- [RetriBooru: Leakage-Free Retrieval of Conditions from Reference Images for Subject-Driven Generation](https://mlsanity.com/paper/2312.02521/) (2023)
- [CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization](https://mlsanity.com/paper/2404.00521/) (2024)
- [ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images](https://mlsanity.com/paper/2502.05928/) (2025)
- [TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learning](https://mlsanity.com/paper/2509.26524/) (2025)
- [Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and Uncurated Unlabeled Data](https://mlsanity.com/paper/2307.08319/) (2023)
- [Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictions](https://mlsanity.com/paper/2401.13174/) (2024)
- [Style Quantization for Data-Efficient GAN Training](https://mlsanity.com/paper/2503.24282/) (2025)
- [Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesis](https://mlsanity.com/paper/2506.21731/) (2025)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) rely heavily on large-scale training data for training high-quality image generation models. With limited training data, the GAN discriminator often suffers from severe overfitting which directly leads to degraded generation especially in generation diversity. Inspired by the recent advances in knowledge distillation (KD), we propose KD-DLGAN, a knowledge-distillation based generation framework that introduces pre-trained vision-language models for training effective data-limited generation models. KD-DLGAN consists of two innovative designs. The first is aggregated generative KD that mitigates the discriminator overfitting by challenging the discriminator with harder learning tasks and distilling more generalizable knowledge from the pre-trained models. The second is correlated generative KD that improves the generation diversity by distilling and preserving the diverse image-text correlation within the pre-trained models. Extensive experiments over multiple benchmarks show that KD-DLGAN achieves superior image generation with limited training data. In addition, KD-DLGAN complements the state-of-the-art with consistent and substantial performance gains.