# Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation

arXiv: https://arxiv.org/abs/2104.15082 · HTML: https://mlsanity.com/paper/2104.15082/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 28 of 28)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation](https://mlsanity.com/paper/1901.02985/) (2019)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [DetNAS: Backbone Search for Object Detection](https://mlsanity.com/paper/1903.10979/) (2019)
- [DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer](https://mlsanity.com/paper/1707.01220/) (2017)
- [Co-Evolutionary Compression for Unpaired Image Translation](https://mlsanity.com/paper/1907.10804/) (2019)
- [Compressing GANs using Knowledge Distillation](https://mlsanity.com/paper/1902.00159/) (2019)

## Built on by (most cited first, 13 of 13)

- [Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation](https://mlsanity.com/paper/2203.06321/) (2022)
- [PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection](https://mlsanity.com/paper/2205.11098/) (2022)
- [Region-aware Knowledge Distillation for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2205.12451/) (2022)
- [Discriminator-Cooperated Feature Map Distillation for GAN Compression](https://mlsanity.com/paper/2212.14169/) (2022)
- [Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection](https://mlsanity.com/paper/2211.08398/) (2022)
- [DISC: Latent Diffusion Models with Self-Distillation from Separated Conditions for Prostate Cancer Grading](https://mlsanity.com/paper/2404.13097/) (2024)
- [Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images](https://mlsanity.com/paper/2501.09268/) (2025)
- [Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Data](https://mlsanity.com/paper/2504.09899/) (2025)
- [Learning to Structure an Image with Few Colors and Beyond](https://mlsanity.com/paper/2208.08438/) (2022)
- [Exploring Content Relationships for Distilling Efficient GANs](https://mlsanity.com/paper/2212.11091/) (2022)
- [Semantic-Supervised Spatial-Temporal Fusion for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2503.10579/) (2025)
- [Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study](https://mlsanity.com/paper/2305.13232/) (2023)
- [Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models](https://mlsanity.com/paper/2309.14928/) (2023)

## Abstract (arXiv)

Generative adversarial networks (GANs) have shown significant potential in modeling high dimensional distributions of image data, especially on image-to-image translation tasks. However, due to the complexity of these tasks, state-of-the-art models often contain a tremendous amount of parameters, which results in large model size and long inference time. In this work, we propose a novel method to address this problem by applying knowledge distillation together with distillation of a semantic relation preserving matrix. This matrix, derived from the teacher's feature encoding, helps the student model learn better semantic relations. In contrast to existing compression methods designed for classification tasks, our proposed method adapts well to the image-to-image translation task on GANs. Experiments conducted on 5 different datasets and 3 different pairs of teacher and student models provide strong evidence that our methods achieve impressive results both qualitatively and quantitatively.