# The Spatially-Correlative Loss for Various Image Translation Tasks

arXiv: https://arxiv.org/abs/2104.00854 · HTML: https://mlsanity.com/paper/2104.00854/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 59 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 30 of 48)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [DualGAN: Unsupervised Dual Learning for Image-to-Image Translation](https://mlsanity.com/paper/1704.02510/) (2017)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)

## Built on by (most cited first, 30 of 66)

- [EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations](https://mlsanity.com/paper/2207.06635/) (2022)
- [Unpaired Image-to-Image Translation via Neural Schr"odinger Bridge](https://mlsanity.com/paper/2305.15086/) (2023)
- [Relational Embedding for Few-Shot Classification](https://mlsanity.com/paper/2108.09666/) (2021)
- [StyleAlign: Analysis and Applications of Aligned StyleGAN Models](https://mlsanity.com/paper/2110.11323/) (2021)
- [Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Tasks](https://mlsanity.com/paper/2203.01532/) (2022)
- [QS-Attn: Query-Selected Attention for Contrastive Learning in I2I Translation](https://mlsanity.com/paper/2203.08483/) (2022)
- [ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank](https://mlsanity.com/paper/2312.06135/) (2023)
- [Look Closer to Supervise Better: One-Shot Font Generation via Component-Based Discriminator](https://mlsanity.com/paper/2205.00146/) (2022)
- [General Image-to-Image Translation with One-Shot Image Guidance](https://mlsanity.com/paper/2307.14352/) (2023)
- [InstaFormer: Instance-Aware Image-to-Image Translation with Transformer](https://mlsanity.com/paper/2203.16248/) (2022)
- [Structure Representation Network and Uncertainty Feedback Learning for Dense Non-Uniform Fog Removal](https://mlsanity.com/paper/2210.03061/) (2022)
- [Modulated Contrast for Versatile Image Synthesis](https://mlsanity.com/paper/2203.09333/) (2022)
- [Partially Does It: Towards Scene-Level FG-SBIR with Partial Input](https://mlsanity.com/paper/2203.14804/) (2022)
- [Marginal Contrastive Correspondence for Guided Image Generation](https://mlsanity.com/paper/2204.00442/) (2022)
- [Robustness via Uncertainty-aware Cycle Consistency](https://mlsanity.com/paper/2110.12467/) (2021)
- [Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt](https://mlsanity.com/paper/2404.11474/) (2024)
- [ITTR: Unpaired Image-to-Image Translation with Transformers](https://mlsanity.com/paper/2203.16015/) (2022)
- [SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2308.02154/) (2023)
- [Spectral Normalization and Dual Contrastive Regularization for Image-to-Image Translation](https://mlsanity.com/paper/2304.11319/) (2023)
- [Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models](https://mlsanity.com/paper/2309.01590/) (2023)
- [Exploring Semantic Consistency in Unpaired Image Translation to Generate Data for Surgical Applications](https://mlsanity.com/paper/2309.03048/) (2023)
- [Maximum Spatial Perturbation Consistency for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2203.12707/) (2022)
- [UVCGAN v2: An Improved Cycle-Consistent GAN for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2303.16280/) (2023)
- [C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with Confidence-Guided Reliable Object Generation](https://mlsanity.com/paper/2411.10788/) (2024)
- [Rethinking the Paradigm of Content Constraints in Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2211.10867/) (2022)
- [CycleDiff: Cycle Diffusion Models for Unpaired Image-to-image Translation](https://mlsanity.com/paper/2508.06625/) (2025)
- [ManiFest: Manifold Deformation for Few-shot Image Translation](https://mlsanity.com/paper/2111.13681/) (2021)
- [Exploring Negatives in Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2204.11018/) (2022)
- [ReGeneration Learning of Diffusion Models with Rich Prompts for Zero-Shot Image Translation](https://mlsanity.com/paper/2305.04651/) (2023)
- [Every Pixel Has its Moments: Ultra-High-Resolution Unpaired Image-to-Image Translation via Dense Normalization](https://mlsanity.com/paper/2407.04245/) (2024)

## Abstract (arXiv)

We propose a novel spatially-correlative loss that is simple, efficient and yet effective for preserving scene structure consistency while supporting large appearance changes during unpaired image-to-image (I2I) translation. Previous methods attempt this by using pixel-level cycle-consistency or feature-level matching losses, but the domain-specific nature of these losses hinder translation across large domain gaps. To address this, we exploit the spatial patterns of self-similarity as a means of defining scene structure. Our spatially-correlative loss is geared towards only capturing spatial relationships within an image rather than domain appearance. We also introduce a new self-supervised learning method to explicitly learn spatially-correlative maps for each specific translation task. We show distinct improvement over baseline models in all three modes of unpaired I2I translation: single-modal, multi-modal, and even single-image translation. This new loss can easily be integrated into existing network architectures and thus allows wide applicability.