# Compressing GANs using Knowledge Distillation

arXiv: https://arxiv.org/abs/1902.00159 · HTML: https://mlsanity.com/paper/1902.00159/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2019.

## Built on by (most cited first, 30 of 31)

- [Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation](https://mlsanity.com/paper/2105.08919/) (2021)
- [BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping](https://mlsanity.com/paper/2306.05544/) (2023)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Anycost GANs for Interactive Image Synthesis and Editing](https://mlsanity.com/paper/2103.03243/) (2021)
- [Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models](https://mlsanity.com/paper/2211.02048/) (2022)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [Content-Aware GAN Compression](https://mlsanity.com/paper/2104.02244/) (2021)
- [Online Multi-Granularity Distillation for GAN Compression](https://mlsanity.com/paper/2108.06908/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [MFAGAN: A Compression Framework for Memory-Efficient On-Device Super-Resolution GAN](https://mlsanity.com/paper/2107.12679/) (2021)
- [Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation](https://mlsanity.com/paper/2104.15082/) (2021)
- [The Devil is in the GAN: Backdoor Attacks and Defenses in Deep Generative Models](https://mlsanity.com/paper/2108.01644/) (2021)
- [Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis](https://mlsanity.com/paper/2207.05049/) (2022)
- [MobileStyleGAN: A Lightweight Convolutional Neural Network for High-Fidelity Image Synthesis](https://mlsanity.com/paper/2104.04767/) (2021)
- [Mind the Gap in Distilling StyleGANs](https://mlsanity.com/paper/2208.08840/) (2022)
- [Generative Adversarial Super-Resolution at the Edge with Knowledge Distillation](https://mlsanity.com/paper/2209.03355/) (2022)
- [Discriminator-Cooperated Feature Map Distillation for GAN Compression](https://mlsanity.com/paper/2212.14169/) (2022)
- [CFTS-GAN: Continual Few-Shot Teacher Student for Generative Adversarial Networks](https://mlsanity.com/paper/2410.14749/) (2024)
- [SKDCGN: Source-free Knowledge Distillation of Counterfactual Generative Networks using cGANs](https://mlsanity.com/paper/2208.04226/) (2022)
- [Evolutionary Generative Adversarial Networks with Crossover Based Knowledge Distillation](https://mlsanity.com/paper/2101.11186/) (2021)
- [PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors](https://mlsanity.com/paper/2203.04314/) (2022)
- [PPCD-GAN: Progressive Pruning and Class-Aware Distillation for Large-Scale Conditional GANs Compression](https://mlsanity.com/paper/2203.08456/) (2022)
- [CoroNetGAN: Controlled Pruning of GANs via Hypernetworks](https://mlsanity.com/paper/2403.08261/) (2024)
- [Evaluating Post-Training Compression in GANs using Locality-Sensitive Hashing](https://mlsanity.com/paper/2103.11912/) (2021)
- [Exploring Content Relationships for Distilling Efficient GANs](https://mlsanity.com/paper/2212.11091/) (2022)
- [A Stepwise Distillation Learning Strategy for Non-differentiable Visual Programming Frameworks on Visual Reasoning Tasks](https://mlsanity.com/paper/2309.09809/) (2023)
- [Accelerating Inference of Discrete Autoregressive Normalizing Flows by Selective Jacobi Decoding](https://mlsanity.com/paper/2505.24791/) (2025)
- [A Survey on GAN Acceleration Using Memory Compression Technique](https://mlsanity.com/paper/2108.06626/) (2021)
- [Transferring Knowledge with Attention Distillation for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/2108.07466/) (2021)
- [IE-GAN: An Improved Evolutionary Generative Adversarial Network Using a New Fitness Function and a Generic Crossover Operator](https://mlsanity.com/paper/2109.11078/) (2021)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardware, such as mobile devices, and for applications with real time capabilities. There has been no work found to reduce the number of parameters used in GANs. Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller "student" GAN learns to mimic a larger "teacher" GAN. We show that the distillation methods used on MNIST, CIFAR-10, and Celeb-A datasets can compress teacher GANs at ratios of 1669:1, 58:1, and 87:1, respectively, while retaining the quality of the generated image. From our experiments, we observe a qualitative limit for GAN's compression. Moreover, we observe that, with a fixed parameter budget, compressed GANs outperform GANs trained using standard training methods. We conjecture that this is partially owing to the optimization landscape of over-parameterized GANs which allows efficient training using alternating gradient descent. Thus, training an over-parameterized GAN followed by our proposed compression scheme provides a high quality generative model with a small number of parameters.