# Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis

arXiv: https://arxiv.org/abs/1910.06809 · HTML: https://mlsanity.com/paper/1910.06809/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2019.

## Built on by (most cited first, 30 of 108)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [BBDM: Image-to-image Translation with Brownian Bridge Diffusion Models](https://mlsanity.com/paper/2205.07680/) (2022)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Dress Code: High-Resolution Multi-Category Virtual Try-On](https://mlsanity.com/paper/2204.08532/) (2022)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [Pixel-wise Anomaly Detection in Complex Driving Scenes](https://mlsanity.com/paper/2103.05445/) (2021)
- [GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds](https://mlsanity.com/paper/2104.07659/) (2021)
- [Multimodal Conditional Image Synthesis with Product-of-Experts GANs](https://mlsanity.com/paper/2112.05130/) (2021)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [Freestyle Layout-to-Image Synthesis](https://mlsanity.com/paper/2303.14412/) (2023)
- [SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing](https://mlsanity.com/paper/2112.02236/) (2021)
- [Text to Image Generation with Semantic-Spatial Aware GAN](https://mlsanity.com/paper/2104.00567/) (2021)
- [Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data](https://mlsanity.com/paper/2111.06849/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [StyleAlign: Analysis and Applications of Aligned StyleGAN Models](https://mlsanity.com/paper/2110.11323/) (2021)
- [Image Synthesis via Semantic Composition](https://mlsanity.com/paper/2109.07053/) (2021)
- [Learning Semantic Person Image Generation by Region-Adaptive Normalization](https://mlsanity.com/paper/2104.06650/) (2021)
- [Modeling Image Composition for Complex Scene Generation](https://mlsanity.com/paper/2206.00923/) (2022)
- [Context-Aware Layout to Image Generation with Enhanced Object Appearance](https://mlsanity.com/paper/2103.11897/) (2021)
- [Generative Semantic Communication: Diffusion Models Beyond Bit Recovery](https://mlsanity.com/paper/2306.04321/) (2023)
- [MobileFaceSwap: A Lightweight Framework for Video Face Swapping](https://mlsanity.com/paper/2201.03808/) (2022)
- [High-Fidelity Guided Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2211.17084/) (2022)
- [Learned Spatial Representations for Few-shot Talking-Head Synthesis](https://mlsanity.com/paper/2104.14557/) (2021)
- [Dual Contrastive Loss and Attention for GANs](https://mlsanity.com/paper/2103.16748/) (2021)
- [SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation](https://mlsanity.com/paper/2308.10156/) (2023)
- [Cross-View Panorama Image Synthesis](https://mlsanity.com/paper/2203.11832/) (2022)
- [Collaging Class-specific GANs for Semantic Image Synthesis](https://mlsanity.com/paper/2110.04281/) (2021)
- [Compositional Transformers for Scene Generation](https://mlsanity.com/paper/2111.08960/) (2021)
- [Generated Distributions Are All You Need for Membership Inference Attacks Against Generative Models](https://mlsanity.com/paper/2310.19410/) (2023)

## Abstract (arXiv)

Semantic image synthesis aims at generating photorealistic images from semantic layouts. Previous approaches with conditional generative adversarial networks (GAN) show state-of-the-art performance on this task, which either feed the semantic label maps as inputs to the generator, or use them to modulate the activations in normalization layers via affine transformations. We argue that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images. In order to better exploit the semantic layout for the image generator, we propose to predict convolutional kernels conditioned on the semantic label map to generate the intermediate feature maps from the noise maps and eventually generate the images. Moreover, we propose a feature pyramid semantics-embedding discriminator, which is more effective in enhancing fine details and semantic alignments between the generated images and the input semantic layouts than previous multi-scale discriminators. We achieve state-of-the-art results on both quantitative metrics and subjective evaluation on various semantic segmentation datasets, demonstrating the effectiveness of our approach.