mlsanityData to 7 Oct 2026

Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation

arXiv 2203.06321 ↗First posted 12 Mar 2022cs.CV cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Remarkable achievements have been attained with Generative Adversarial Networks (GANs) in image-to-image translation. However, due to a tremendous amount of parameters, state-of-the-art GANs usually suffer from low efficiency and bulky memory usage. To tackle this challenge, firstly, this paper investigates GANs performance from a frequency perspective. The results show that GANs, especially small GANs lack the ability to generate high-quality high frequency information. To address this problem, we propose a novel knowledge distillation method referred to as wavelet knowledge distillation. Instead of directly distilling the generated images of teachers, wavelet knowledge distillation first decomposes the images into different frequency bands with discrete wavelet transformation and then only distills the high frequency bands. As a result, the student GAN can pay more attention to its learning on high frequency bands. Experiments demonstrate that our method leads to 7.08 times compression and 6.80 times acceleration on CycleGAN with almost no performance drop. Additionally, we have studied the relation between discriminators and generators which shows that the compression of discriminators can promote the performance of compressed generators.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 12 Mar 2022.

Builds on

43 of 58 references.

References · 43

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  10. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  12. 2019Relational Knowledge Distillationcited by 873
31 more
  1. 2019Contrastive Representation Distillationcited by 734
  2. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  3. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  4. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  5. 2017Deep Mutual Learningcited by 621
  6. 2019Similarity-Preserving Knowledge Distillationcited by 507
  7. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  8. 2016Generating Videos with Scene Dynamicscited by 376
  9. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  10. 2019Variational Information Distillation for Knowledge Transfercited by 321
  11. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  12. 2018Multi-level Wavelet-CNN for Image Restorationcited by 184
  13. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  14. 2021ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flowscited by 113
  15. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  16. 2021Image-to-image Translation via Hierarchical Style Disentanglementcited by 63
  17. 2021Rethinking and Improving the Robustness of Image Style Transfercited by 48
  18. 2019Breaking the cycle -- Colleagues are all you needcited by 40
  19. 2021Teachers Do More Than Teach: Compressing Image-to-Image Modelscited by 37
  20. 2020Distilling portable Generative Adversarial Networks for Image Translationcited by 36
  21. 2018Attention-GAN for Object Transfiguration in Wild Imagescited by 32
  22. 2021Style-Aware Normalized Loss for Improving Arbitrary Style Transfercited by 30
  23. 2021Content-Aware GAN Compressioncited by 28
  24. 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
  25. 2021Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Schemecited by 23
  26. 2019SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Modelscited by 22
  27. 2017Wavelet Convolutional Neural Networks for Texture Classificationcited by 21
  28. 2021Memory-guided Unsupervised Image-to-image Translationcited by 21
  29. 2021Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translationcited by 13
  30. 2017Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networkscited by 8
  31. 2019TKD: Temporal Knowledge Distillation for Active Perceptioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 11; 2024: 7; 2025: 9; 2026: 4.

39 papers.

Sort:

By area:

  1. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  2. 2023FreeKD: Knowledge Distillation via Semantic Frequency Promptcited by 27
  3. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  4. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  5. 2024MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolutioncited by 17
  6. 2022FreGAN: Exploiting Frequency Components for Training GANs under Limited Datacited by 16
  7. 2023MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillationcited by 16
  8. 2023BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusioncited by 13
  9. 2022Region-aware Knowledge Distillation for Efficient Image-to-Image Translationcited by 10
  10. 2022Label-Guided Auxiliary Training Improves 3D Object Detectorcited by 6
  11. 2023One-for-All: Towards Universal Domain Translation with a Single StyleGANcited by 6
  12. 2024Diversity-aware Channel Pruning for StyleGAN Compressioncited by 5
27 more
  1. 2024Estimating Depth of Monocular Panoramic Image with Teacher-Student Model Fusing Equirectangular and Spherical Representationscited by 5
  2. 2025DWTGS: Rethinking Frequency Regularization for Sparse-view 3D Gaussian Splattingcited by 5
  3. 2022PCT-CycleGAN: Paired Complementary Temporal Cycle-Consistent Adversarial Networks for Radar-Based Precipitation Nowcastingcited by 4
  4. 2022Discriminator-Cooperated Feature Map Distillation for GAN Compressioncited by 4
  5. 2022Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detectioncited by 3
  6. 2023UGC: Unified GAN Compression for Efficient Image-to-Image Translationcited by 3
  7. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
  8. 2025Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Imagescited by 3
  9. 2026Enhancing Indoor Occupancy Prediction via Sparse Query-Based Multi-Level Consistent Knowledge Distillationcited by 3
  10. 2023Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentationcited by 2
  11. 2024CoroNetGAN: Controlled Pruning of GANs via Hypernetworkscited by 2
  12. 2024Nickel and Diming Your GAN: A Dual-Method Approach to Enhancing GAN Efficiency via Knowledge Distillationcited by 2
  13. 2025Soft Knowledge Distillation with Multi-Dimensional Cross-Net Attention for Image Restoration Models Compressioncited by 2
  14. 2025Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Datacited by 2
  15. 2022Exploring Content Relationships for Distilling Efficient GANscited by 1
  16. 2023Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environmentscited by 1
  17. 2025Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Reviewcited by 1
  18. 2025Distilled-3DGS:Distilled 3D Gaussian Splattingcited by 1
  19. 2025Dual-Stream Spectral Decoupling Distillation for Remote Sensing Object Detectioncited by 1
  20. 2026Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentationcited by 1
  21. 2023Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Studycited by 0
  22. 2023Both Spatial and Frequency Cues Contribute to High-Fidelity Image Inpaintingcited by 0
  23. 2024Neural-Network-Enhanced Metalens Camera for High-Definition, Dynamic Imaging in the Long-Wave Infrared Spectrumcited by 0
  24. 2025FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfercited by 0
  25. 2025SDAKD: Student Discriminator Assisted Knowledge Distillation for Super-Resolution Generative Adversarial Networkscited by 0
  26. 2026FreqKD: Frequency-Decoupled Cross-Modal Knowledge Distillation for Infrared Object Detectioncited by 0
  27. 2026Physics-Guided Spectral Distillation for Underwater Image Enhancement on Resource-Constrained Devicescited by 0