# Online Multi-Granularity Distillation for GAN Compression

arXiv: https://arxiv.org/abs/2108.06908 · HTML: https://mlsanity.com/paper/2108.06908/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 76 references that are in the map.
- First posted on arXiv 16 Aug 2021.

## Builds on (most cited first, 30 of 49)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)

## Built on by (most cited first, 24 of 24)

- [Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis](https://mlsanity.com/paper/2404.13686/) (2024)
- [FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation](https://mlsanity.com/paper/2303.17225/) (2023)
- [PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection](https://mlsanity.com/paper/2205.11098/) (2022)
- [Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2112.08996/) (2021)
- [Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme](https://mlsanity.com/paper/2110.14439/) (2021)
- [BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion](https://mlsanity.com/paper/2305.15798/) (2023)
- [Region-aware Knowledge Distillation for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2205.12451/) (2022)
- [Diversity-aware Channel Pruning for StyleGAN Compression](https://mlsanity.com/paper/2403.13548/) (2024)
- [Discriminator-Cooperated Feature Map Distillation for GAN Compression](https://mlsanity.com/paper/2212.14169/) (2022)
- [Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection](https://mlsanity.com/paper/2211.08398/) (2022)
- [UGC: Unified GAN Compression for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2309.09310/) (2023)
- [Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold](https://mlsanity.com/paper/2312.14776/) (2023)
- [Information-Theoretic GAN Compression with Variational Energy-based Model](https://mlsanity.com/paper/2303.16050/) (2023)
- [Shortcut-V2V: Compression Framework for Video-to-Video Translation based on Temporal Redundancy Reduction](https://mlsanity.com/paper/2308.08011/) (2023)
- [CoroNetGAN: Controlled Pruning of GANs via Hypernetworks](https://mlsanity.com/paper/2403.08261/) (2024)
- [CSHNet: A Novel Information Asymmetric Image Translation Method](https://mlsanity.com/paper/2501.10197/) (2025)
- [Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillation](https://mlsanity.com/paper/2507.18352/) (2025)
- [Exploring Content Relationships for Distilling Efficient GANs](https://mlsanity.com/paper/2212.11091/) (2022)
- [DGL-GAN: Discriminator Guided Learning for GAN Compression](https://mlsanity.com/paper/2112.06502/) (2021)
- [A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation](https://mlsanity.com/paper/2304.00471/) (2023)
- [Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study](https://mlsanity.com/paper/2305.13232/) (2023)
- [Physics-informed generative real-time lens-free imaging](https://mlsanity.com/paper/2403.07786/) (2024)
- [SDAKD: Student Discriminator Assisted Knowledge Distillation for Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/2510.03870/) (2025)
- [Improving Generative Adversarial Networks with Self-Distillation](https://mlsanity.com/paper/2605.08577/) (2026)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) have witnessed prevailing success in yielding outstanding images, however, they are burdensome to deploy on resource-constrained devices due to ponderous computational costs and hulking memory usage. Although recent efforts on compressing GANs have acquired remarkable results, they still exist potential model redundancies and can be further compressed. To solve this issue, we propose a novel online multi-granularity distillation (OMGD) scheme to obtain lightweight GANs, which contributes to generating high-fidelity images with low computational demands. We offer the first attempt to popularize single-stage online distillation for GAN-oriented compression, where the progressively promoted teacher generator helps to refine the discriminator-free based student generator. Complementary teacher generators and network layers provide comprehensive and multi-granularity concepts to enhance visual fidelity from diverse dimensions. Experimental results on four benchmark datasets demonstrate that OMGD successes to compress 40x MACs and 82.5X parameters on Pix2Pix and CycleGAN, without loss of image quality. It reveals that OMGD provides a feasible solution for the deployment of real-time image translation on resource-constrained devices. Our code and models are made public at: https://github.com/bytedance/OMGD.