# DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs

arXiv: https://arxiv.org/abs/2011.05867 · HTML: https://mlsanity.com/paper/2011.05867/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2020.

## Built on by (most cited first, 11 of 11)

- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Robustness via Uncertainty-aware Cycle Consistency](https://mlsanity.com/paper/2110.12467/) (2021)
- [Facial-Sketch Synthesis: A New Challenge](https://mlsanity.com/paper/2112.15439/) (2021)
- [ISF-GAN: An Implicit Style Function for High-Resolution Image-to-Image Translation](https://mlsanity.com/paper/2109.12492/) (2021)
- [TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets](https://mlsanity.com/paper/2105.06219/) (2021)
- [Controllable Unlearning for Image-to-Image Generative Models via varepsilon-Constrained Optimization](https://mlsanity.com/paper/2408.01689/) (2024)
- [Transferring Unconditional to Conditional GANs with Hyper-Modulation](https://mlsanity.com/paper/2112.02219/) (2021)
- [3D-Aware Multi-Class Image-to-Image Translation with NeRFs](https://mlsanity.com/paper/2303.15012/) (2023)
- [UTSGAN: Unseen Transition Suss GAN for Transition-Aware Image-to-image Translation](https://mlsanity.com/paper/2304.11955/) (2023)
- [Independent Encoder for Deep Hierarchical Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2107.02494/) (2021)
- [Structure-Preserving Unpaired Image Translation to Photometrically Calibrate JunoCam with Hubble Data](https://mlsanity.com/paper/2511.22668/) (2025)

## Abstract (arXiv)

Image-to-image translation has recently achieved remarkable results. But despite current success, it suffers from inferior performance when translations between classes require large shape changes. We attribute this to the high-resolution bottlenecks which are used by current state-of-the-art image-to-image methods. Therefore, in this work, we propose a novel deep hierarchical Image-to-Image Translation method, called DeepI2I. We learn a model by leveraging hierarchical features: (a) structural information contained in the shallow layers and (b) semantic information extracted from the deep layers. To enable the training of deep I2I models on small datasets, we propose a novel transfer learning method, that transfers knowledge from pre-trained GANs. Specifically, we leverage the discriminator of a pre-trained GANs (i.e. BigGAN or StyleGAN) to initialize both the encoder and the discriminator and the pre-trained generator to initialize the generator of our model. Applying knowledge transfer leads to an alignment problem between the encoder and generator. We introduce an adaptor network to address this. On many-class image-to-image translation on three datasets (Animal faces, Birds, and Foods) we decrease mFID by at least 35% when compared to the state-of-the-art. Furthermore, we qualitatively and quantitatively demonstrate that transfer learning significantly improves the performance of I2I systems, especially for small datasets. Finally, we are the first to perform I2I translations for domains with over 100 classes.