# Modulated Contrast for Versatile Image Synthesis

arXiv: https://arxiv.org/abs/2203.09333 · HTML: https://mlsanity.com/paper/2203.09333/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 43)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning](https://mlsanity.com/paper/2004.11660/) (2020)
- [Contrastive Learning for Compact Single Image Dehazing](https://mlsanity.com/paper/2104.09367/) (2021)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Unsupervised Cross-Domain Image Generation](https://mlsanity.com/paper/1611.02200/) (2016)
- [Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1809.05852/) (2018)
- [One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1706.00826/) (2017)
- [Diverse Image Inpainting with Bidirectional and Autoregressive Transformers](https://mlsanity.com/paper/2104.12335/) (2021)
- [Spatial Fusion GAN for Image Synthesis](https://mlsanity.com/paper/1812.05840/) (2018)

## Built on by (most cited first, 23 of 23)

- [Regularized Vector Quantization for Tokenized Image Synthesis](https://mlsanity.com/paper/2303.06424/) (2023)
- [Marginal Contrastive Correspondence for Guided Image Generation](https://mlsanity.com/paper/2204.00442/) (2022)
- [Towards Counterfactual Image Manipulation via CLIP](https://mlsanity.com/paper/2207.02812/) (2022)
- [VMRF: View Matching Neural Radiance Fields](https://mlsanity.com/paper/2207.02621/) (2022)
- [Auto-regressive Image Synthesis with Integrated Quantization](https://mlsanity.com/paper/2207.10776/) (2022)
- [Rethinking the Paradigm of Content Constraints in Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2211.10867/) (2022)
- [TPSeNCE: Towards Artifact-Free Realistic Rain Generation for Deraining and Object Detection in Rain](https://mlsanity.com/paper/2311.00660/) (2023)
- [CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis](https://mlsanity.com/paper/2211.14286/) (2022)
- [Energy-Guided Diffusion Model for CBCT-to-CT Synthesis](https://mlsanity.com/paper/2308.03354/) (2023)
- [Patch-wise Graph Contrastive Learning for Image Translation](https://mlsanity.com/paper/2312.08223/) (2023)
- [USIGAN: Unbalanced Self-Information Feature Transport for Weakly Paired Image IHC Virtual Staining](https://mlsanity.com/paper/2507.05843/) (2025)
- [Contrastive Learning and Cycle Consistency-based Transductive Transfer Learning for Target Annotation](https://mlsanity.com/paper/2401.12340/) (2024)
- [Deformation-aware GAN for Medical Image Synthesis with Substantially Misaligned Pairs](https://mlsanity.com/paper/2408.09432/) (2024)
- [MGHF: Multi-Granular High-Frequency Perceptual Loss for Image Super-Resolution](https://mlsanity.com/paper/2411.13548/) (2024)
- [Patch-Wise Hypergraph Contrastive Learning with Dual Normal Distribution Weighting for Multi-Domain Stain Transfer](https://mlsanity.com/paper/2503.09523/) (2025)
- [Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Data](https://mlsanity.com/paper/2504.09899/) (2025)
- [A 3D Multi-Style Cross-Modality Segmentation Framework for Segmenting Vestibular Schwannoma and Cochlea](https://mlsanity.com/paper/2311.11578/) (2023)
- [How Good Are Deep Generative Models for Solving Inverse Problems?](https://mlsanity.com/paper/2312.12691/) (2023)
- [Image-to-Image Translation Framework Embedded with Rotation Symmetry Priors](https://mlsanity.com/paper/2604.12805/) (2026)
- [Latent Multi-Relation Reasoning for GAN-Prior based Image Super-Resolution](https://mlsanity.com/paper/2208.02861/) (2022)
- [Disentangled Contrastive Image Translation for Nighttime Surveillance](https://mlsanity.com/paper/2307.05038/) (2023)
- [Hierarchical Conditional Semi-Paired Image-to-Image Translation For Multi-Task Image Defect Correction On Shopping Websites](https://mlsanity.com/paper/2309.05883/) (2023)
- [WarpI2I: Image Warping for Image-to-Image Translation](https://mlsanity.com/paper/2606.31018/) (2026)

## Abstract (arXiv)

Perceiving the similarity between images has been a long-standing and fundamental problem underlying various visual generation tasks. Predominant approaches measure the inter-image distance by computing pointwise absolute deviations, which tends to estimate the median of instance distributions and leads to blurs and artifacts in the generated images. This paper presents MoNCE, a versatile metric that introduces image contrast to learn a calibrated metric for the perception of multifaceted inter-image distances. Unlike vanilla contrast which indiscriminately pushes negative samples from the anchor regardless of their similarity, we propose to re-weight the pushing force of negative samples adaptively according to their similarity to the anchor, which facilitates the contrastive learning from informative negative samples. Since multiple patch-level contrastive objectives are involved in image distance measurement, we introduce optimal transport in MoNCE to modulate the pushing force of negative samples collaboratively across multiple contrastive objectives. Extensive experiments over multiple image translation tasks show that the proposed MoNCE outperforms various prevailing metrics substantially. The code is available at https://github.com/fnzhan/MoNCE.