# Compositional Scene Representation Learning via Reconstruction: A Survey

arXiv: https://arxiv.org/abs/2202.07135 · HTML: https://mlsanity.com/paper/2202.07135/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 81 of its 155 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 30 of 81)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [GRF: Learning a General Radiance Field for 3D Representation and Rendering](https://mlsanity.com/paper/2010.04595/) (2020)
- [NeRF--: Neural Radiance Fields Without Known Camera Parameters](https://mlsanity.com/paper/2102.07064/) (2021)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [NeRF-VAE: A Geometry Aware 3D Scene Generative Model](https://mlsanity.com/paper/2104.00587/) (2021)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)

## Built on by (most cited first, 27 of 27)

- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis](https://mlsanity.com/paper/2306.08645/) (2023)
- [Provable Compositional Generalization for Object-Centric Learning](https://mlsanity.com/paper/2310.05327/) (2023)
- [Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2210.02324/) (2022)
- [Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes](https://mlsanity.com/paper/2305.11772/) (2023)
- [Sensitivity of Slot-Based Object-Centric Models to their Number of Slots](https://mlsanity.com/paper/2305.18890/) (2023)
- [GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians](https://mlsanity.com/paper/2410.01535/) (2024)
- [OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer](https://mlsanity.com/paper/2603.14401/) (2026)
- [Learning Object Permanence from Videos via Latent Imaginations](https://mlsanity.com/paper/2310.10372/) (2023)
- [Benchmarking and Analysis of Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2312.04947/) (2023)
- [Unsupervised Object-Centric Learning from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2401.01922/) (2024)
- [unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning](https://mlsanity.com/paper/2506.01778/) (2025)
- [3D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigation](https://mlsanity.com/paper/2605.26500/) (2026)
- [Compositional Law Parsing with Latent Random Functions](https://mlsanity.com/paper/2209.09115/) (2022)
- [Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction](https://mlsanity.com/paper/2301.08951/) (2023)
- [Object-Aware Query Perturbation for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2407.12346/) (2024)
- [Learning Global Object-Centric Representations via Disentangled Slot Attention](https://mlsanity.com/paper/2410.18809/) (2024)
- [OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning](https://mlsanity.com/paper/2306.09682/) (2023)
- [Towards Generative Abstract Reasoning: Completing Raven's Progressive Matrix via Rule Abstraction and Selection](https://mlsanity.com/paper/2401.09966/) (2024)
- [Disentangling Visual Priors: Unsupervised Learning of Scene Interpretations with Compositional Autoencoder](https://mlsanity.com/paper/2409.09716/) (2024)
- [Compositional Scene Modeling with Global Object-Centric Representations](https://mlsanity.com/paper/2211.11500/) (2022)
- [Ablation Study to Clarify the Mechanism of Object Segmentation in Multi-Object Representation Learning](https://mlsanity.com/paper/2310.03273/) (2023)
- [Loci-Segmented: Improving Scene Segmentation Learning](https://mlsanity.com/paper/2310.10410/) (2023)
- [Improving Viewpoint-Independent Object-Centric Representations through Active Viewpoint Selection](https://mlsanity.com/paper/2411.00402/) (2024)
- [Autoassociative Learning of Structural Representations for Modeling and Classification in Medical Imaging](https://mlsanity.com/paper/2411.12070/) (2024)
- [Multi-Part Object Representations via Graph Structures and Co-Part Discovery](https://mlsanity.com/paper/2512.18192/) (2025)

## Abstract (arXiv)

Visual scenes are composed of visual concepts and have the property of combinatorial explosion. An important reason for humans to efficiently learn from diverse visual scenes is the ability of compositional perception, and it is desirable for artificial intelligence to have similar abilities. Compositional scene representation learning is a task that enables such abilities. In recent years, various methods have been proposed to apply deep neural networks, which have been proven to be advantageous in representation learning, to learn compositional scene representations via reconstruction, advancing this research direction into the deep learning era. Learning via reconstruction is advantageous because it may utilize massive unlabeled data and avoid costly and laborious data annotation. In this survey, we first outline the current progress on reconstruction-based compositional scene representation learning with deep neural networks, including development history and categorizations of existing methods from the perspectives of the modeling of visual scenes and the inference of scene representations; then provide benchmarks, including an open source toolbox to reproduce the benchmark experiments, of representative methods that consider the most extensively studied problem setting and form the foundation for other methods; and finally discuss the limitations of existing methods and future directions of this research topic.