# Diverse Semantic Image Synthesis via Probability Distribution Modeling

arXiv: https://arxiv.org/abs/2103.06878 · HTML: https://mlsanity.com/paper/2103.06878/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 52 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)

## Built on by (most cited first, 30 of 49)

- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [HairCLIP: Design Your Hair by Text and Reference Image](https://mlsanity.com/paper/2112.05142/) (2021)
- [FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models](https://mlsanity.com/paper/2310.15160/) (2023)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Generative Semantic Communication: Diffusion Models Beyond Bit Recovery](https://mlsanity.com/paper/2306.04321/) (2023)
- [EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2401.04608/) (2024)
- [Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model](https://mlsanity.com/paper/2406.17998/) (2024)
- [Reduce Information Loss in Transformers for Pluralistic Image Inpainting](https://mlsanity.com/paper/2205.05076/) (2022)
- [E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion](https://mlsanity.com/paper/2104.07661/) (2021)
- [Scalable Multi-Temporal Remote Sensing Change Data Generation via Simulating Stochastic Change Process](https://mlsanity.com/paper/2309.17031/) (2023)
- [SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow](https://mlsanity.com/paper/2405.20282/) (2024)
- [NASDM: Nuclei-Aware Semantic Histopathology Image Generation Using Diffusion Models](https://mlsanity.com/paper/2303.11477/) (2023)
- [HairCLIPv2: Unifying Hair Editing via Proxy Feature Blending](https://mlsanity.com/paper/2310.10651/) (2023)
- [Realistic Full-Body Anonymization with Surface-Guided GANs](https://mlsanity.com/paper/2201.02193/) (2022)
- [Stochastic Conditional Diffusion Models for Robust Semantic Image Synthesis](https://mlsanity.com/paper/2402.16506/) (2024)
- [SCAM! Transferring humans between images with Semantic Cross Attention Modulation](https://mlsanity.com/paper/2210.04883/) (2022)
- [HairFastGAN: Realistic and Robust Hair Transfer with a Fast Encoder-Based Approach](https://mlsanity.com/paper/2404.01094/) (2024)
- [Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation](https://mlsanity.com/paper/2306.00914/) (2023)
- [Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis](https://mlsanity.com/paper/2307.12084/) (2023)
- [EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models](https://mlsanity.com/paper/2410.09802/) (2024)
- [SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis](https://mlsanity.com/paper/2403.09638/) (2024)
- [DRAN: Detailed Region-Adaptive Normalization for Conditional Image Synthesis](https://mlsanity.com/paper/2109.14525/) (2021)
- [MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier](https://mlsanity.com/paper/2209.11549/) (2022)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints](https://mlsanity.com/paper/2103.15812/) (2021)
- [Automatic Generation of Semantic Parts for Face Image Synthesis](https://mlsanity.com/paper/2307.05317/) (2023)
- [Semantic Image Synthesis via Class-Adaptive Cross-Attention](https://mlsanity.com/paper/2308.16071/) (2023)
- [Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models](https://mlsanity.com/paper/2212.00793/) (2022)
- [Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation](https://mlsanity.com/paper/2309.00216/) (2023)
- [CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation](https://mlsanity.com/paper/2501.02786/) (2025)
- [Mixture of Global and Local Experts with Diffusion Transformer for Controllable Face Generation](https://mlsanity.com/paper/2509.00428/) (2025)

## Abstract (arXiv)

Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level multimodal results, still remains a challenge. In this paper, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at semantic or even instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Code will be available at urlhttps://github.com/tzt101/INADE.git