# Diversifying Semantic Image Synthesis and Editing via Class- and Layer-wise VAEs

arXiv: https://arxiv.org/abs/2106.13416 · HTML: https://mlsanity.com/paper/2106.13416/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [HDR image reconstruction from a single exposure using deep CNNs](https://mlsanity.com/paper/1710.07480/) (2017)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [Diverse Image Synthesis from Semantic Layouts via Conditional IMLE](https://mlsanity.com/paper/1811.12373/) (2018)
- [Multi-Agent Diverse Generative Adversarial Networks](https://mlsanity.com/paper/1704.02906/) (2017)
- [Semantically Multi-modal Image Synthesis](https://mlsanity.com/paper/2003.12697/) (2020)
- [Implicit Maximum Likelihood Estimation](https://mlsanity.com/paper/1809.09087/) (2018)
- [FaceShop: Deep Sketch-based Face Image Editing](https://mlsanity.com/paper/1804.08972/) (2018)

## Built on by (most cited first, 1 of 1)

- [SCAM! Transferring humans between images with Semantic Cross Attention Modulation](https://mlsanity.com/paper/2210.04883/) (2022)

## Abstract (arXiv)

Semantic image synthesis is a process for generating photorealistic images from a single semantic mask. To enrich the diversity of multimodal image synthesis, previous methods have controlled the global appearance of an output image by learning a single latent space. However, a single latent code is often insufficient for capturing various object styles because object appearance depends on multiple factors. To handle individual factors that determine object styles, we propose a class- and layer-wise extension to the variational autoencoder (VAE) framework that allows flexible control over each object class at the local to global levels by learning multiple latent spaces. Furthermore, we demonstrate that our method generates images that are both plausible and more diverse compared to state-of-the-art methods via extensive experiments with real and synthetic datasets inthree different domains. We also show that our method enables a wide range of applications in image synthesis and editing tasks.