# Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation

arXiv: https://arxiv.org/abs/2203.06321 · HTML: https://mlsanity.com/paper/2203.06321/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 12 Mar 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 43)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Multi-level Wavelet-CNN for Image Restoration](https://mlsanity.com/paper/1805.07071/) (2018)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [BERT-of-Theseus: Compressing BERT by Progressive Module Replacing](https://mlsanity.com/paper/2002.02925/) (2020)
- [ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows](https://mlsanity.com/paper/2103.16877/) (2021)
- [Image-to-image Translation via Hierarchical Style Disentanglement](https://mlsanity.com/paper/2103.01456/) (2021)
- [Rethinking and Improving the Robustness of Image Style Transfer](https://mlsanity.com/paper/2104.05623/) (2021)
- [Breaking the cycle -- Colleagues are all you need](https://mlsanity.com/paper/1911.10538/) (2019)

## Built on by (most cited first, 30 of 39)

- [PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection](https://mlsanity.com/paper/2205.11098/) (2022)
- [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023)
- [VkD: Improving Knowledge Distillation using Orthogonal Projections](https://mlsanity.com/paper/2403.06213/) (2024)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution](https://mlsanity.com/paper/2404.09571/) (2024)
- [FreGAN: Exploiting Frequency Components for Training GANs under Limited Data](https://mlsanity.com/paper/2210.05461/) (2022)
- [MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation](https://mlsanity.com/paper/2303.07815/) (2023)
- [BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion](https://mlsanity.com/paper/2305.15798/) (2023)
- [Region-aware Knowledge Distillation for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2205.12451/) (2022)
- [Label-Guided Auxiliary Training Improves 3D Object Detector](https://mlsanity.com/paper/2207.11753/) (2022)
- [One-for-All: Towards Universal Domain Translation with a Single StyleGAN](https://mlsanity.com/paper/2310.14222/) (2023)
- [DWTGS: Rethinking Frequency Regularization for Sparse-view 3D Gaussian Splatting](https://mlsanity.com/paper/2507.15690/) (2025)
- [Diversity-aware Channel Pruning for StyleGAN Compression](https://mlsanity.com/paper/2403.13548/) (2024)
- [Estimating Depth of Monocular Panoramic Image with Teacher-Student Model Fusing Equirectangular and Spherical Representations](https://mlsanity.com/paper/2405.16858/) (2024)
- [PCT-CycleGAN: Paired Complementary Temporal Cycle-Consistent Adversarial Networks for Radar-Based Precipitation Nowcasting](https://mlsanity.com/paper/2211.15046/) (2022)
- [Discriminator-Cooperated Feature Map Distillation for GAN Compression](https://mlsanity.com/paper/2212.14169/) (2022)
- [Enhancing Indoor Occupancy Prediction via Sparse Query-Based Multi-Level Consistent Knowledge Distillation](https://mlsanity.com/paper/2602.02318/) (2026)
- [Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection](https://mlsanity.com/paper/2211.08398/) (2022)
- [UGC: Unified GAN Compression for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2309.09310/) (2023)
- [RdimKD: Generic Distillation Paradigm by Dimensionality Reduction](https://mlsanity.com/paper/2312.08700/) (2023)
- [Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images](https://mlsanity.com/paper/2501.09268/) (2025)
- [Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2304.14800/) (2023)
- [CoroNetGAN: Controlled Pruning of GANs via Hypernetworks](https://mlsanity.com/paper/2403.08261/) (2024)
- [Nickel and Diming Your GAN: A Dual-Method Approach to Enhancing GAN Efficiency via Knowledge Distillation](https://mlsanity.com/paper/2405.11614/) (2024)
- [Soft Knowledge Distillation with Multi-Dimensional Cross-Net Attention for Image Restoration Models Compression](https://mlsanity.com/paper/2501.09321/) (2025)
- [Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Data](https://mlsanity.com/paper/2504.09899/) (2025)
- [Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentation](https://mlsanity.com/paper/2605.26382/) (2026)
- [Exploring Content Relationships for Distilling Efficient GANs](https://mlsanity.com/paper/2212.11091/) (2022)
- [Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments](https://mlsanity.com/paper/2311.05970/) (2023)
- [Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2505.07866/) (2025)

## Abstract (arXiv)

Remarkable achievements have been attained with Generative Adversarial Networks (GANs) in image-to-image translation. However, due to a tremendous amount of parameters, state-of-the-art GANs usually suffer from low efficiency and bulky memory usage. To tackle this challenge, firstly, this paper investigates GANs performance from a frequency perspective. The results show that GANs, especially small GANs lack the ability to generate high-quality high frequency information. To address this problem, we propose a novel knowledge distillation method referred to as wavelet knowledge distillation. Instead of directly distilling the generated images of teachers, wavelet knowledge distillation first decomposes the images into different frequency bands with discrete wavelet transformation and then only distills the high frequency bands. As a result, the student GAN can pay more attention to its learning on high frequency bands. Experiments demonstrate that our method leads to 7.08 times compression and 6.80 times acceleration on CycleGAN with almost no performance drop. Additionally, we have studied the relation between discriminators and generators which shows that the compression of discriminators can promote the performance of compressed generators.