# Efficient Semantic Image Synthesis via Class-Adaptive Normalization

arXiv: https://arxiv.org/abs/2012.04644 · HTML: https://mlsanity.com/paper/2012.04644/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 30 of 58)

- [Flow Matching in Latent Space](https://mlsanity.com/paper/2307.08698/) (2023)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [HairCLIP: Design Your Hair by Text and Reference Image](https://mlsanity.com/paper/2112.05142/) (2021)
- [Freestyle Layout-to-Image Synthesis](https://mlsanity.com/paper/2303.14412/) (2023)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [SceneComposer: Any-Level Semantic Image Synthesis](https://mlsanity.com/paper/2211.11742/) (2022)
- [Reduce Information Loss in Transformers for Pluralistic Image Inpainting](https://mlsanity.com/paper/2205.05076/) (2022)
- [E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion](https://mlsanity.com/paper/2104.07661/) (2021)
- [DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation](https://mlsanity.com/paper/2207.06124/) (2022)
- [Arbitrary-Scale Image Synthesis](https://mlsanity.com/paper/2204.02273/) (2022)
- [NASDM: Nuclei-Aware Semantic Histopathology Image Generation Using Diffusion Models](https://mlsanity.com/paper/2303.11477/) (2023)
- [Hierarchical Diffusion Autoencoders and Disentangled Image Manipulation](https://mlsanity.com/paper/2304.11829/) (2023)
- [Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive](https://mlsanity.com/paper/2401.08815/) (2024)
- [HairCLIPv2: Unifying Hair Editing via Proxy Feature Blending](https://mlsanity.com/paper/2310.10651/) (2023)
- [Realistic Full-Body Anonymization with Surface-Guided GANs](https://mlsanity.com/paper/2201.02193/) (2022)
- [Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis](https://mlsanity.com/paper/2204.02854/) (2022)
- [Modality Cycles with Masked Conditional Diffusion for Unsupervised Anomaly Segmentation in MRI](https://mlsanity.com/paper/2308.16150/) (2023)
- [Stochastic Conditional Diffusion Models for Robust Semantic Image Synthesis](https://mlsanity.com/paper/2402.16506/) (2024)
- [Context-Consistent Semantic Image Editing with Style-Preserved Modulation](https://mlsanity.com/paper/2207.06252/) (2022)
- [SCAM! Transferring humans between images with Semantic Cross Attention Modulation](https://mlsanity.com/paper/2210.04883/) (2022)
- [HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach](https://mlsanity.com/paper/2404.01094/) (2024)
- [FashionTex: Controllable Virtual Try-on with Text and Texture](https://mlsanity.com/paper/2305.04451/) (2023)
- [Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation](https://mlsanity.com/paper/2306.00914/) (2023)
- [Learning 3D Photography Videos via Self-supervised Diffusion on Single Images](https://mlsanity.com/paper/2302.10781/) (2023)
- [Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis](https://mlsanity.com/paper/2307.12084/) (2023)
- [EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models](https://mlsanity.com/paper/2410.09802/) (2024)
- [Location-aware Adaptive Normalization: A Deep Learning Approach For Wildfire Danger Forecasting](https://mlsanity.com/paper/2212.08208/) (2022)
- [Enhancing Object Coherence in Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.10522/) (2023)
- [SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis](https://mlsanity.com/paper/2403.09638/) (2024)

## Abstract (arXiv)

Spatially-adaptive normalization (SPADE) is remarkably successful recently in conditional semantic image synthesis citepark2019semantic, which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to prevent the semantic information from being washed away. Despite its impressive performance, a more thorough understanding of the advantages inside the box is still highly demanded to help reduce the significant computation and parameter overhead introduced by this novel structure. In this paper, from a return-on-investment point of view, we conduct an in-depth analysis of the effectiveness of this spatially-adaptive normalization and observe that its modulation parameters benefit more from semantic-awareness rather than spatial-adaptiveness, especially for high-resolution input masks. Inspired by this observation, we propose class-adaptive normalization (CLADE), a lightweight but equally-effective variant that is only adaptive to semantic class. In order to further improve spatial-adaptiveness, we introduce intra-class positional map encoding calculated from semantic layouts to modulate the normalization parameters of CLADE and propose a truly spatially-adaptive variant of CLADE, namely CLADE-ICPE.Through extensive experiments on multiple challenging datasets, we demonstrate that the proposed CLADE can be generalized to different SPADE-based methods while achieving comparable generation quality compared to SPADE, but it is much more efficient with fewer extra parameters and lower computational cost. The code and pretrained models are available at urlhttps://github.com/tzt101/CLADE.git.