# Rethinking the Paradigm of Content Constraints in Unpaired Image-to-Image Translation

arXiv: https://arxiv.org/abs/2211.10867 · HTML: https://mlsanity.com/paper/2211.10867/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 20 Nov 2022.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks](https://mlsanity.com/paper/1809.00437/) (2018)
- [Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1809.05852/) (2018)
- [FD-GAN: Generative Adversarial Networks with Fusion-discriminator for Single Image Dehazing](https://mlsanity.com/paper/2001.06968/) (2020)
- [The Spatially-Correlative Loss for Various Image Translation Tasks](https://mlsanity.com/paper/2104.00854/) (2021)
- [Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2003.00273/) (2020)

## Built on by (most cited first, 7 of 7)

- [USIGAN: Unbalanced Self-Information Feature Transport for Weakly Paired Image IHC Virtual Staining](https://mlsanity.com/paper/2507.05843/) (2025)
- [Patch-Wise Hypergraph Contrastive Learning with Dual Normal Distribution Weighting for Multi-Domain Stain Transfer](https://mlsanity.com/paper/2503.09523/) (2025)
- [UDAPose: Unsupervised Domain Adaptation for Low-Light Human Pose Estimation](https://mlsanity.com/paper/2604.10485/) (2026)
- [Self Identity Mapping](https://mlsanity.com/paper/2509.18165/) (2025)
- [A Framework for Low-Effort Training Data Generation for Urban Semantic Segmentation](https://mlsanity.com/paper/2510.11567/) (2025)
- [FlowForm: Synergizing Fluid Physics with Topological Consistency for Satellite Flood Synthesis](https://mlsanity.com/paper/2608.03822/) (2026)
- [Through the Schr"odinger Bridge: Benchmarking Antemortem Image Restoration from Postmortem Autolysis to Enhance Forensic Diagnostics](https://mlsanity.com/paper/2608.21813/) (2026)

## Abstract (arXiv)

In an unpaired setting, lacking sufficient content constraints for image-to-image translation (I2I) tasks, GAN-based approaches are usually prone to model collapse. Current solutions can be divided into two categories, reconstruction-based and Siamese network-based. The former requires that the transformed or transforming image can be perfectly converted back to the original image, which is sometimes too strict and limits the generative performance. The latter involves feeding the original and generated images into a feature extractor and then matching their outputs. This is not efficient enough, and a universal feature extractor is not easily available. In this paper, we propose EnCo, a simple but efficient way to maintain the content by constraining the representational similarity in the latent space of patch-level features from the same stage of the Encoder and deCoder of the generator. For the similarity function, we use a simple MSE loss instead of contrastive loss, which is currently widely used in I2I tasks. Benefits from the design, EnCo training is extremely efficient, while the features from the encoder produce a more positive effect on the decoding, leading to more satisfying generations. In addition, we rethink the role played by discriminators in sampling patches and propose a discriminative attention-guided (DAG) patch sampling strategy to replace random sampling. DAG is parameter-free and only requires negligible computational overhead, while significantly improving the performance of the model. Extensive experiments on multiple datasets demonstrate the effectiveness and advantages of EnCo, and we achieve multiple state-of-the-art compared to previous methods. Our code is available at https://github.com/XiudingCai/EnCo-pytorch.