# Semantic Bottleneck Scene Generation

arXiv: https://arxiv.org/abs/1911.11357 · HTML: https://mlsanity.com/paper/1911.11357/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 19 of 19)

- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [Compositional Transformers for Scene Generation](https://mlsanity.com/paper/2111.08960/) (2021)
- [Semantic Palette: Guiding Scene Generation with Class Proportions](https://mlsanity.com/paper/2106.01629/) (2021)
- [GD-VDM: Generated Depth for better Diffusion-based Video Generation](https://mlsanity.com/paper/2306.11173/) (2023)
- [DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotation](https://mlsanity.com/paper/2403.06168/) (2024)
- [Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis](https://mlsanity.com/paper/2308.08157/) (2023)
- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [Co-synthesis of Histopathology Nuclei Image-Label Pairs using a Context-Conditioned Joint Diffusion Model](https://mlsanity.com/paper/2407.14434/) (2024)
- [SemIE: Semantically-aware Image Extrapolation](https://mlsanity.com/paper/2108.13702/) (2021)
- [Few-shot Semantic Image Synthesis with Class Affinity Transfer](https://mlsanity.com/paper/2304.02321/) (2023)
- [Towards Pragmatic Semantic Image Synthesis for Urban Scenes](https://mlsanity.com/paper/2305.09726/) (2023)
- [Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis](https://mlsanity.com/paper/2410.21638/) (2024)
- [Urban-StyleGAN: Learning to Generate and Manipulate Images of Urban Scenes](https://mlsanity.com/paper/2305.09602/) (2023)
- [Learning to Generate Training Datasets for Robust Semantic Segmentation](https://mlsanity.com/paper/2308.02535/) (2023)
- [Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects](https://mlsanity.com/paper/2410.01816/) (2024)
- [MeronymNet: A Hierarchical Approach for Unified and Controllable Multi-Category Object Generation](https://mlsanity.com/paper/2110.08818/) (2021)
- [Unifying conditional and unconditional semantic image synthesis with OCO-GAN](https://mlsanity.com/paper/2211.14105/) (2022)
- [Data-driven Camera and Lidar Simulation Models for Autonomous Driving: A Review from Generative Models to Volume Renderers](https://mlsanity.com/paper/2402.10079/) (2024)

## Abstract (arXiv)

Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes. We assume pixel-wise segmentation labels are available during training and use them to learn the scene structure. During inference, our model first synthesizes a realistic segmentation layout from scratch, then synthesizes a realistic scene conditioned on that layout. For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts. For the latter, we use a conditional segmentation-to-image synthesis network that captures the distribution of photo-realistic images conditioned on the semantic layout. When trained end-to-end, the resulting model outperforms state-of-the-art generative models in unsupervised image synthesis on two challenging domains in terms of the Frechet Inception Distance and user-study evaluations. Moreover, we demonstrate the generated segmentation maps can be used as additional training data to strongly improve recent segmentation-to-image synthesis networks.