# Transferring Knowledge with Attention Distillation for Multi-Domain Image-to-Image Translation

arXiv: https://arxiv.org/abs/2108.07466 · HTML: https://mlsanity.com/paper/2108.07466/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 30 of 30)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Towards Visually Explaining Variational Autoencoders](https://mlsanity.com/paper/1911.07389/) (2019)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes](https://mlsanity.com/paper/1908.07269/) (2019)
- [TextureGAN: Controlling Deep Image Synthesis with Texture Patches](https://mlsanity.com/paper/1706.02823/) (2017)
- [Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization](https://mlsanity.com/paper/1908.06965/) (2019)
- [Compressing GANs using Knowledge Distillation](https://mlsanity.com/paper/1902.00159/) (2019)
- [MetalGAN: Multi-Domain Label-Less Image Synthesis Using cGANs and Meta-Learning](https://mlsanity.com/paper/1912.02494/) (2019)

## Abstract (arXiv)

Gradient-based attention modeling has been used widely as a way to visualize and understand convolutional neural networks. However, exploiting these visual explanations during the training of generative adversarial networks (GANs) is an unexplored area in computer vision research. Indeed, we argue that this kind of information can be used to influence GANs training in a positive way. For this reason, in this paper, it is shown how gradient based attentions can be used as knowledge to be conveyed in a teacher-student paradigm for multi-domain image-to-image translation tasks in order to improve the results of the student architecture. Further, it is demonstrated how "pseudo"-attentions can also be employed during training when teacher and student networks are trained on different domains which share some similarities. The approach is validated on multi-domain facial attributes transfer and human expression synthesis showing both qualitative and quantitative results.