# Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis

arXiv: https://arxiv.org/abs/2204.02854 · HTML: https://mlsanity.com/paper/2204.02854/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [You Only Need Adversarial Supervision for Semantic Image Synthesis](https://mlsanity.com/paper/2012.04781/) (2020)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation](https://mlsanity.com/paper/1912.12215/) (2019)
- [Diverse Image Synthesis from Semantic Layouts via Conditional IMLE](https://mlsanity.com/paper/1811.12373/) (2018)
- [Efficient Semantic Image Synthesis via Class-Adaptive Normalization](https://mlsanity.com/paper/2012.04644/) (2020)
- [Semi-parametric Image Synthesis](https://mlsanity.com/paper/1804.10992/) (2018)
- [Image Synthesis via Semantic Composition](https://mlsanity.com/paper/2109.07053/) (2021)
- [SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects](https://mlsanity.com/paper/2004.04977/) (2020)
- [Learning Hierarchical Semantic Image Manipulation through Structured Representations](https://mlsanity.com/paper/1808.07535/) (2018)
- [Collaging Class-specific GANs for Semantic Image Synthesis](https://mlsanity.com/paper/2110.04281/) (2021)

## Built on by (most cited first, 15 of 15)

- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [Freestyle Layout-to-Image Synthesis](https://mlsanity.com/paper/2303.14412/) (2023)
- [Can segmentation models be trained with fully synthetically generated data?](https://mlsanity.com/paper/2209.08256/) (2022)
- [Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model](https://mlsanity.com/paper/2406.17998/) (2024)
- [Scalable Multi-Temporal Remote Sensing Change Data Generation via Simulating Stochastic Change Process](https://mlsanity.com/paper/2309.17031/) (2023)
- [PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis](https://mlsanity.com/paper/2403.01852/) (2024)
- [Stochastic Conditional Diffusion Models for Robust Semantic Image Synthesis](https://mlsanity.com/paper/2402.16506/) (2024)
- [Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis](https://mlsanity.com/paper/2307.12084/) (2023)
- [Enhancing Object Coherence in Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.10522/) (2023)
- [SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis](https://mlsanity.com/paper/2403.09638/) (2024)
- [Semantic Image Synthesis via Class-Adaptive Cross-Attention](https://mlsanity.com/paper/2308.16071/) (2023)
- [FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exits](https://mlsanity.com/paper/2304.10306/) (2023)
- [Inferring and Leveraging Parts from Object Shape for Improving Semantic Image Synthesis](https://mlsanity.com/paper/2305.19547/) (2023)
- [LaMI-GO: Latent Mixture Integration for Goal-Oriented Communications Achieving High Spectrum Efficiency](https://mlsanity.com/paper/2412.17839/) (2024)
- [Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding](https://mlsanity.com/paper/2411.14781/) (2024)

## Abstract (arXiv)

Semantic image synthesis is a challenging task with many practical applications. Albeit remarkable progress has been made in semantic image synthesis with spatially-adaptive normalization and existing methods normalize the feature activations under the coarse-level guidance (e.g., semantic class). However, different parts of a semantic object (e.g., wheel and window of car) are quite different in structures and textures, making blurry synthesis results usually inevitable due to the missing of fine-grained guidance. In this paper, we propose a novel normalization module, termed as REtrieval-based Spatially AdaptIve normaLization (RESAIL), for introducing pixel level fine-grained guidance to the normalization architecture. Specifically, we first present a retrieval paradigm by finding a content patch of the same semantic class from training set with the most similar shape to each test semantic mask. Then, RESAIL is presented to use the retrieved patch for guiding the feature normalization of corresponding region, and can provide pixel level fine-grained guidance, thereby greatly mitigating blurry synthesis results. Moreover, distorted ground-truth images are also utilized as alternatives of retrieval-based guidance for feature normalization, further benefiting model training and improving visual quality of generated images. Experiments on several challenging datasets show that our RESAIL performs favorably against state-of-the-arts in terms of quantitative metrics, visual quality, and subjective evaluation. The source code and pre-trained models will be publicly available.