mlsanityData to 7 Oct 2026

Online Multi-Granularity Distillation for GAN Compression

arXiv 2108.06908 ↗First posted 16 Aug 2021cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Generative Adversarial Networks (GANs) have witnessed prevailing success in yielding outstanding images, however, they are burdensome to deploy on resource-constrained devices due to ponderous computational costs and hulking memory usage. Although recent efforts on compressing GANs have acquired remarkable results, they still exist potential model redundancies and can be further compressed. To solve this issue, we propose a novel online multi-granularity distillation (OMGD) scheme to obtain lightweight GANs, which contributes to generating high-fidelity images with low computational demands. We offer the first attempt to popularize single-stage online distillation for GAN-oriented compression, where the progressively promoted teacher generator helps to refine the discriminator-free based student generator. Complementary teacher generators and network layers provide comprehensive and multi-granularity concepts to enhance visual fidelity from diverse dimensions. Experimental results on four benchmark datasets demonstrate that OMGD successes to compress 40x MACs and 82.5X parameters on Pix2Pix and CycleGAN, without loss of image quality. It reveals that OMGD provides a feasible solution for the deployment of real-time image translation on resource-constrained devices. Our code and models are made public at: https://github.com/bytedance/OMGD.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 49 of its 76 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

49 of 76 references.

References · 49

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2016Improved Techniques for Training GANscited by 2,938
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
37 more
  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  3. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  4. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  5. 2018Self-Attention Generative Adversarial Networkscited by 809
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  7. 2016Generative Adversarial Text to Image Synthesiscited by 735
  8. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  9. 2017Wasserstein GANcited by 635
  10. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  11. 2015A Neural Algorithm of Artistic Stylecited by 480
  12. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  13. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  14. 2018Slimmable Neural Networkscited by 301
  15. 2017Dilated Residual Networkscited by 293
  16. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  17. 2018Large scale distributed neural network training through online distillationcited by 144
  18. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  19. 2019Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translationcited by 97
  20. 2020Bringing Old Photos Back to Lifecited by 95
  21. 2019AutoGAN: Neural Architecture Search for Generative Adversarial Networkscited by 94
  22. 2016High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesiscited by 89
  23. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  24. 2020AutoGAN-Distiller: Searching to Compress Generative Adversarial Networkscited by 50
  25. 2020PAMS: Quantized Super-Resolution via Parameterized Max Scalecited by 46
  26. 2020GAN Slimming: All-in-One GAN Compression by A Unified Optimization Frameworkcited by 44
  27. 2021Teachers Do More Than Teach: Compressing Image-to-Image Modelscited by 37
  28. 2020Distilling portable Generative Adversarial Networks for Image Translationcited by 36
  29. 2019Co-Evolutionary Compression for Unpaired Image Translationcited by 32
  30. 2019Compressing GANs using Knowledge Distillationcited by 31
  31. 2020Slimmable Generative Adversarial Networkscited by 27
  32. 2020Channel Distillation: Channel-Wise Attention for Knowledge Distillationcited by 20
  33. 2019OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networkscited by 18
  34. 2021Network Pruning using Adaptive Exemplar Filterscited by 16
  35. 2017Building Fast and Compact Convolutional Neural Networks for Offline Handwritten Chinese Character Recognitioncited by 14
  36. 2020TinyGAN: Distilling BigGAN for Conditional Image Generationcited by 14
  37. 2017Design of a Very Compact CNN Classifier for Online Handwritten Chinese Character Recognition Using DropWeight and Global Poolingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 8; 2024: 4; 2025: 3; 2026: 1.

24 papers.

Sort:

By area:

  1. 2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 153
  2. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 94
  3. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  4. 2021Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentationcited by 29
  5. 2021Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Schemecited by 23
  6. 2023BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusioncited by 13
  7. 2022Region-aware Knowledge Distillation for Efficient Image-to-Image Translationcited by 10
  8. 2024Diversity-aware Channel Pruning for StyleGAN Compressioncited by 5
  9. 2022Discriminator-Cooperated Feature Map Distillation for GAN Compressioncited by 4
  10. 2022Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detectioncited by 3
  11. 2023UGC: Unified GAN Compression for Efficient Image-to-Image Translationcited by 3
  12. 2023Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifoldcited by 3
12 more
  1. 2023Information-Theoretic GAN Compression with Variational Energy-based Modelcited by 2
  2. 2023Shortcut-V2V: Compression Framework for Video-to-Video Translation based on Temporal Redundancy Reductioncited by 2
  3. 2024CoroNetGAN: Controlled Pruning of GANs via Hypernetworkscited by 2
  4. 2025CSHNet: A Novel Information Asymmetric Image Translation Methodcited by 2
  5. 2025Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillationcited by 2
  6. 2022Exploring Content Relationships for Distilling Efficient GANscited by 1
  7. 2021DGL-GAN: Discriminator Guided Learning for GAN Compressioncited by 0
  8. 2023A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generationcited by 0
  9. 2023Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Studycited by 0
  10. 2024Physics-informed generative real-time lens-free imagingcited by 0
  11. 2025SDAKD: Student Discriminator Assisted Knowledge Distillation for Super-Resolution Generative Adversarial Networkscited by 0
  12. 2026Improving Generative Adversarial Networks with Self-Distillationcited by 0