mlsanityData to 7 Oct 2026

Region-aware Knowledge Distillation for Efficient Image-to-Image Translation

arXiv 2205.12451 ↗First posted 25 May 2022cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent progress in image-to-image translation has witnessed the success of generative adversarial networks (GANs). However, GANs usually contain a huge number of parameters, which lead to intolerant memory and computation consumption and limit their deployment on edge devices. To address this issue, knowledge distillation is proposed to transfer the knowledge from a cumbersome teacher model to an efficient student model. However, most previous knowledge distillation methods are designed for image classification and lead to limited performance in image-to-image translation. In this paper, we propose Region-aware Knowledge Distillation ReKo to compress image-to-image translation models. Firstly, ReKo adaptively finds the crucial regions in the images with an attention module. Then, patch-wise contrastive learning is adopted to maximize the mutual information between students and teachers in these crucial regions. Experiments with eight comparison methods on nine datasets demonstrate the substantial effectiveness of ReKo on both paired and unpaired image-to-image translation. For instance, our 7.08X compressed and 6.80X accelerated CycleGAN student outperforms its teacher by 1.33 and 1.04 FID scores on Horse to Zebra and Zebra to Horse, respectively. Codes will be released on GitHub.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 63 of its 76 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

63 of 76 references.

References · 63

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
38 more
  1. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  5. 2014Conditional Generative Adversarial Netscited by 2,152
  6. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,250
  10. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  12. 2019Relational Knowledge Distillationcited by 873
  13. 2019Contrastive Representation Distillationcited by 734
  14. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  15. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  16. 2017Deep Mutual Learningcited by 621
  17. 2019Similarity-Preserving Knowledge Distillationcited by 507
  18. 2018Video-to-Video Synthesiscited by 388
  19. 2017DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networkscited by 366
  20. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  21. 2019Variational Information Distillation for Knowledge Transfercited by 321
  22. 2017DualGAN: Unsupervised Dual Learning for Image-to-Image Translationcited by 310
  23. 2017Learning to Discover Cross-Domain Relations with Generative Adversarial Networkscited by 278
  24. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  25. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  26. 2021ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flowscited by 113
  27. 2019AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networkscited by 77
  28. 2020Wasserstein Contrastive Representation Distillationcited by 63
  29. 2021Image-to-image Translation via Hierarchical Style Disentanglementcited by 63
  30. 2018Unsupervised Attention-guided Image to Image Translationcited by 58
  31. 2020Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillationcited by 51
  32. 2021Rethinking and Improving the Robustness of Image Style Transfercited by 48
  33. 2020GAN Slimming: All-in-One GAN Compression by A Unified Optimization Frameworkcited by 44
  34. 2019SPA-GAN: Spatial Attention GAN for Image-to-Image Translationcited by 43
  35. 2021Complementary Relation Contrastive Distillationcited by 42
  36. 2021Rethinking Style Transfer: From Pixels to Parameterized Brushstrokescited by 42
  37. 2019Attention-Guided Generative Adversarial Networks for Unsupervised Image-to-Image Translationcited by 39
  38. 2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 2.

10 papers.

  1. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  3. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  5. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  6. 2022Label-Guided Auxiliary Training Improves 3D Object Detectorcited by 6
  7. 2022Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detectioncited by 3
  8. 2023Masked Discriminators for Content-Consistent Unpaired Image-to-Image Translationcited by 2
  9. 2024CoroNetGAN: Controlled Pruning of GANs via Hypernetworkscited by 2
  10. 2023Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representationscited by 0