# Unbalanced Feature Transport for Exemplar-based Image Translation

arXiv: https://arxiv.org/abs/2106.10482 · HTML: https://mlsanity.com/paper/2106.10482/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 68 references that are in the map.
- First posted on arXiv 19 Jun 2021.

## Builds on (most cited first, 30 of 56)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [Controllable Person Image Synthesis with Attribute-Decomposed GAN](https://mlsanity.com/paper/2003.12267/) (2020)
- [Optimal Entropy-Transport problems and a new Hellinger-Kantorovich distance between positive measures](https://mlsanity.com/paper/1508.07941/) (2015)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Cross-domain Correspondence Learning for Exemplar-based Image Translation](https://mlsanity.com/paper/2004.05571/) (2020)
- [Image Generation from Layout](https://mlsanity.com/paper/1811.11389/) (2018)
- [Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translation](https://mlsanity.com/paper/1904.06807/) (2019)

## Built on by (most cited first, 30 of 41)

- [Optimal Flow Matching: Learning Straight Trajectories in Just One Step](https://mlsanity.com/paper/2403.13117/) (2024)
- [Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16223/) (2023)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [WaveFill: A Wavelet-based Generation Network for Image Inpainting](https://mlsanity.com/paper/2107.11027/) (2021)
- [Blind Image Super-Resolution via Contrastive Representation Learning](https://mlsanity.com/paper/2107.00708/) (2021)
- [Eliminating Gradient Conflict in Reference-based Line-Art Colorization](https://mlsanity.com/paper/2207.06095/) (2022)
- [DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation](https://mlsanity.com/paper/2207.06124/) (2022)
- [Modulated Contrast for Versatile Image Synthesis](https://mlsanity.com/paper/2203.09333/) (2022)
- [MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation](https://mlsanity.com/paper/2209.11047/) (2022)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [Marginal Contrastive Correspondence for Guided Image Generation](https://mlsanity.com/paper/2204.00442/) (2022)
- [SOOD: Towards Semi-Supervised Oriented Object Detection](https://mlsanity.com/paper/2304.04515/) (2023)
- [Sparse Needlets for Lighting Estimation with Spherical Transport Loss](https://mlsanity.com/paper/2106.13090/) (2021)
- [VMRF: View Matching Neural Radiance Fields](https://mlsanity.com/paper/2207.02621/) (2022)
- [KD-DLGAN: Data Limited Image Generation via Knowledge Distillation](https://mlsanity.com/paper/2303.17158/) (2023)
- [GenCo: Generative Co-training for Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2110.01254/) (2021)
- [Face Transformer: Towards High Fidelity and Accurate Face Swapping](https://mlsanity.com/paper/2304.02530/) (2023)
- [Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition](https://mlsanity.com/paper/2408.12475/) (2024)
- [Auto-regressive Image Synthesis with Integrated Quantization](https://mlsanity.com/paper/2207.10776/) (2022)
- [EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models](https://mlsanity.com/paper/2410.09802/) (2024)
- [Unsupervised Cross-Domain Image Retrieval via Prototypical Optimal Transport](https://mlsanity.com/paper/2402.18411/) (2024)
- [ManiFest: Manifold Deformation for Few-shot Image Translation](https://mlsanity.com/paper/2111.13681/) (2021)
- [Exploring Negatives in Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2204.11018/) (2022)
- [TPSeNCE: Towards Artifact-Free Realistic Rain Generation for Deraining and Object Detection in Rain](https://mlsanity.com/paper/2311.00660/) (2023)
- [Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation](https://mlsanity.com/paper/2309.00216/) (2023)
- [MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification](https://mlsanity.com/paper/2502.07409/) (2025)
- [USIGAN: Unbalanced Self-Information Feature Transport for Weakly Paired Image IHC Virtual Staining](https://mlsanity.com/paper/2507.05843/) (2025)
- [FBC-GAN: Diverse and Flexible Image Synthesis via Foreground-Background Composition](https://mlsanity.com/paper/2107.03166/) (2021)
- [CFFT-GAN: Cross-domain Feature Fusion Transformer for Exemplar-based Image Translation](https://mlsanity.com/paper/2302.01608/) (2023)
- [Unpaired Multi-domain Attribute Translation of 3D Facial Shapes with a Square and Symmetric Geometric Map](https://mlsanity.com/paper/2308.13245/) (2023)

## Abstract (arXiv)

Despite the great success of GANs in images translation with different conditioned inputs such as semantic segmentation and edge maps, generating high-fidelity realistic images with reference styles remains a grand challenge in conditional image-to-image translation. This paper presents a general image translation framework that incorporates optimal transport for feature alignment between conditional inputs and style exemplars in image translation. The introduction of optimal transport mitigates the constraint of many-to-one feature matching significantly while building up accurate semantic correspondences between conditional inputs and exemplars. We design a novel unbalanced optimal transport to address the transport between features with deviational distributions which exists widely between conditional inputs and exemplars. In addition, we design a semantic-activation normalization scheme that injects style features of exemplars into the image translation process successfully. Extensive experiments over multiple image translation tasks show that our method achieves superior image translation qualitatively and quantitatively as compared with the state-of-the-art.