# BlobGAN: Spatially Disentangled Scene Representations

arXiv: https://arxiv.org/abs/2205.02837 · HTML: https://mlsanity.com/paper/2205.02837/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 77 of its 106 references that are in the map.
- First posted on arXiv 5 May 2022.

## Builds on (most cited first, 30 of 77)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)

## Built on by (most cited first, 30 of 35)

- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [CC3D: Layout-Conditioned Generation of Compositional 3D Scenes](https://mlsanity.com/paper/2303.12074/) (2023)
- [PanoDiffusion: 360-degree Panorama Outpainting via Diffusion](https://mlsanity.com/paper/2307.03177/) (2023)
- [Rosetta Neurons: Mining the Common Units in a Model Zoo](https://mlsanity.com/paper/2306.09346/) (2023)
- [Perceptual Artifacts Localization for Image Synthesis Tasks](https://mlsanity.com/paper/2310.05590/) (2023)
- [ObjectSDF++: Improved Object-Compositional Neural Implicit Surfaces](https://mlsanity.com/paper/2308.07868/) (2023)
- [OCTET: Object-aware Counterfactual Explanations](https://mlsanity.com/paper/2211.12380/) (2022)
- [Text-to-Image Models for Counterfactual Explanations: a Black-Box Approach](https://mlsanity.com/paper/2309.07944/) (2023)
- [UrbanGIRAFFE: Representing Urban Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2303.14167/) (2023)
- [PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor](https://mlsanity.com/paper/2303.17546/) (2023)
- [A Survey on Deep Generative 3D-aware Image Synthesis](https://mlsanity.com/paper/2210.14267/) (2022)
- [AdaptNet: Policy Adaptation for Physics-Based Character Control](https://mlsanity.com/paper/2310.00239/) (2023)
- [Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis](https://mlsanity.com/paper/2308.02840/) (2023)
- [ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing](https://mlsanity.com/paper/2405.11190/) (2024)
- [Editable Image Elements for Controllable Synthesis](https://mlsanity.com/paper/2404.16029/) (2024)
- [Move Anything with Layered Scene Diffusion](https://mlsanity.com/paper/2404.07178/) (2024)
- [Text2Place: Affordance-aware Text Guided Human Placement](https://mlsanity.com/paper/2407.15446/) (2024)
- [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://mlsanity.com/paper/2503.21770/) (2025)
- [Conditional 360-degree Image Synthesis for Immersive Indoor Scene Decoration](https://mlsanity.com/paper/2307.09621/) (2023)
- [Category-Agnostic Neural Object Rigging](https://mlsanity.com/paper/2505.20283/) (2025)
- [BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes](https://mlsanity.com/paper/2303.14706/) (2023)
- [Controllable Image Generation via Collage Representations](https://mlsanity.com/paper/2304.13722/) (2023)
- [StyLitGAN: Prompting StyleGAN to Produce New Illumination Conditions](https://mlsanity.com/paper/2205.10351/) (2022)
- [Spatial Steerability of GANs via Self-Supervision from Discriminator](https://mlsanity.com/paper/2301.08455/) (2023)
- [GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers](https://mlsanity.com/paper/2411.15605/) (2024)

## Abstract (arXiv)

We propose an unsupervised, mid-level representation for a generative model of scenes. The representation is mid-level in that it is neither per-pixel nor per-image; rather, scenes are modeled as a collection of spatial, depth-ordered "blobs" of features. Blobs are differentiably placed onto a feature grid that is decoded into an image by a generative adversarial network. Due to the spatial uniformity of blobs and the locality inherent to convolution, our network learns to associate different blobs with different entities in a scene and to arrange these blobs to capture scene layout. We demonstrate this emergent behavior by showing that, despite training without any supervision, our method enables applications such as easy manipulation of objects within a scene (e.g., moving, removing, and restyling furniture), creation of feasible scenes given constraints (e.g., plausible rooms with drawers at a particular location), and parsing of real-world images into constituent parts. On a challenging multi-category dataset of indoor scenes, BlobGAN outperforms StyleGAN2 in image quality as measured by FID. See our project page for video results and interactive demo: https://www.dave.ml/blobgan