# Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

arXiv: https://arxiv.org/abs/1911.09267 · HTML: https://mlsanity.com/paper/1911.09267/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2019.

## Built on by (most cited first, 30 of 106)

- [ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement](https://mlsanity.com/paper/2104.02699/) (2021)
- [CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis](https://mlsanity.com/paper/2110.09788/) (2021)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [3D-aware Image Synthesis via Learning Structural and Textural Representations](https://mlsanity.com/paper/2112.10759/) (2021)
- [Only a Matter of Style: Age Transformation Using a Style-Based Regression Model](https://mlsanity.com/paper/2102.02754/) (2021)
- [GAN-Control: Explicitly Controllable GANs](https://mlsanity.com/paper/2101.02477/) (2021)
- [Style-Based Global Appearance Flow for Virtual Try-On](https://mlsanity.com/paper/2204.01046/) (2022)
- [HyperInverter: Improving StyleGAN Inversion via Hypernetwork](https://mlsanity.com/paper/2112.00719/) (2021)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [High-resolution Face Swapping via Latent Semantics Disentanglement](https://mlsanity.com/paper/2203.15958/) (2022)
- [Low-Rank Subspaces in GANs](https://mlsanity.com/paper/2106.04488/) (2021)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [WarpedGANSpace: Finding non-linear RBF paths in GAN latent space](https://mlsanity.com/paper/2109.13357/) (2021)
- [StyleGAN knows Normal, Depth, Albedo, and More](https://mlsanity.com/paper/2306.00987/) (2023)
- [One-Shot Generative Domain Adaptation](https://mlsanity.com/paper/2111.09876/) (2021)
- [Exploring Diffusion Time-steps for Unsupervised Representation Learning](https://mlsanity.com/paper/2401.11430/) (2024)
- [BlobGAN: Spatially Disentangled Scene Representations](https://mlsanity.com/paper/2205.02837/) (2022)
- [EigenGAN: Layer-Wise Eigen-Learning for GANs](https://mlsanity.com/paper/2104.12476/) (2021)
- [State-of-the-Art in the Architecture, Methods and Applications of StyleGAN](https://mlsanity.com/paper/2202.14020/) (2022)
- [Picture that Sketch: Photorealistic Image Generation from Abstract Sketches](https://mlsanity.com/paper/2303.11162/) (2023)
- [3D-Aware Indoor Scene Synthesis with Depth Priors](https://mlsanity.com/paper/2202.08553/) (2022)
- [StyleFusion: A Generative Model for Disentangling Spatial Segments](https://mlsanity.com/paper/2107.07437/) (2021)
- [Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs](https://mlsanity.com/paper/2106.06959/) (2021)
- [Linear Semantics in Generative Adversarial Networks](https://mlsanity.com/paper/2104.00487/) (2021)
- [3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting](https://mlsanity.com/paper/2405.18424/) (2024)
- [High-fidelity GAN Inversion with Padding Space](https://mlsanity.com/paper/2203.11105/) (2022)
- [Improving GAN Equilibrium by Raising Spatial Awareness](https://mlsanity.com/paper/2112.00718/) (2021)
- [Region-Based Semantic Factorization in GANs](https://mlsanity.com/paper/2202.09649/) (2022)
- [High Resolution Face Editing with Masked GAN Latent Code Optimization](https://mlsanity.com/paper/2103.11135/) (2021)
- [Everything is There in Latent Space: Attribute Editing and Attribute Style Manipulation by StyleGAN Latent Space Exploration](https://mlsanity.com/paper/2207.09855/) (2022)

## Abstract (arXiv)

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what generative models have learned inside the deep generative representations and how photo-realistic images are able to be composed of the layer-wise stochasticity introduced in recent GANs. In this work, we show that highly-structured semantic hierarchy emerges as variation factors from synthesizing scenes from the generative representations in state-of-the-art GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with a broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results further suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent variation factors facilitates semantic scene manipulation.