# Semantic Palette: Guiding Scene Generation with Class Proportions

arXiv: https://arxiv.org/abs/2106.01629 · HTML: https://mlsanity.com/paper/2106.01629/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [Modulating early visual processing by language](https://mlsanity.com/paper/1707.00683/) (2017)
- [IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments](https://mlsanity.com/paper/1811.10200/) (2018)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [How good is my GAN?](https://mlsanity.com/paper/1807.09499/) (2018)
- [GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution](https://mlsanity.com/paper/1611.04051/) (2016)
- [Semantic Bottleneck Scene Generation](https://mlsanity.com/paper/1911.11357/) (2019)
- [Conditional Generative Adversarial Networks for Data Augmentation and Adaptation in Remotely Sensed Imagery](https://mlsanity.com/paper/1908.03809/) (2019)

## Built on by (most cited first, 14 of 14)

- [DREAM: Visual Decoding from Reversing Human Visual System](https://mlsanity.com/paper/2310.02265/) (2023)
- [Reliability in Semantic Segmentation: Can We Use Synthetic Data?](https://mlsanity.com/paper/2312.09231/) (2023)
- [DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotation](https://mlsanity.com/paper/2403.06168/) (2024)
- [Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis](https://mlsanity.com/paper/2308.08157/) (2023)
- [Controlled Training Data Generation with Diffusion Models](https://mlsanity.com/paper/2403.15309/) (2024)
- [Co-synthesis of Histopathology Nuclei Image-Label Pairs using a Context-Conditioned Joint Diffusion Model](https://mlsanity.com/paper/2407.14434/) (2024)
- [SAC-GAN: Structure-Aware Image Composition](https://mlsanity.com/paper/2112.06596/) (2021)
- [Towards Pragmatic Semantic Image Synthesis for Urban Scenes](https://mlsanity.com/paper/2305.09726/) (2023)
- [Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis](https://mlsanity.com/paper/2410.21638/) (2024)
- [Urban-StyleGAN: Learning to Generate and Manipulate Images of Urban Scenes](https://mlsanity.com/paper/2305.09602/) (2023)
- [Learning to Generate Training Datasets for Robust Semantic Segmentation](https://mlsanity.com/paper/2308.02535/) (2023)
- [A Semi-Paired Approach For Label-to-Image Translation](https://mlsanity.com/paper/2306.13585/) (2023)
- [Referenceless User Controllable Semantic Image Synthesis](https://mlsanity.com/paper/2306.10646/) (2023)
- [Data-driven Camera and Lidar Simulation Models for Autonomous Driving: A Review from Generative Models to Volume Renderers](https://mlsanity.com/paper/2402.10079/) (2024)

## Abstract (arXiv)

Despite the recent progress of generative adversarial networks (GANs) at synthesizing photo-realistic images, producing complex urban scenes remains a challenging problem. Previous works break down scene generation into two consecutive phases: unconditional semantic layout synthesis and image synthesis conditioned on layouts. In this work, we propose to condition layout generation as well for higher semantic control: given a vector of class proportions, we generate layouts with matching composition. To this end, we introduce a conditional framework with novel architecture designs and learning objectives, which effectively accommodates class proportions to guide the scene generation process. The proposed architecture also allows partial layout editing with interesting applications. Thanks to the semantic control, we can produce layouts close to the real distribution, helping enhance the whole scene generation process. On different metrics and urban scene benchmarks, our models outperform existing baselines. Moreover, we demonstrate the merit of our approach for data augmentation: semantic segmenters trained on real layout-image pairs along with additional ones generated by our approach outperform models only trained on real pairs.