# RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces

arXiv: https://arxiv.org/abs/2007.01272 · HTML: https://mlsanity.com/paper/2007.01272/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 30 of 32)

- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Testing Relational Understanding in Text-Guided Image Generation](https://mlsanity.com/paper/2208.00005/) (2022)
- [SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing](https://mlsanity.com/paper/2112.02236/) (2021)
- [CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields](https://mlsanity.com/paper/2103.17269/) (2021)
- [Unsupervised Object Learning via Common Fate](https://mlsanity.com/paper/2110.06562/) (2021)
- [Compositional Transformers for Scene Generation](https://mlsanity.com/paper/2111.08960/) (2021)
- [Learning What and Where: Disentangling Location and Identity Tracking Without Supervision](https://mlsanity.com/paper/2205.13349/) (2022)
- [Slot-VAE: Object-Centric Scene Generation with Slot Attention](https://mlsanity.com/paper/2306.06997/) (2023)
- [Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field](https://mlsanity.com/paper/2302.01579/) (2023)
- [Relations, Negations, and Numbers: Looking for Logic in Generative Text-to-Image Models](https://mlsanity.com/paper/2411.17066/) (2024)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [Learning Foreground-Background Segmentation from Improved Layered GANs](https://mlsanity.com/paper/2104.00483/) (2021)
- [Panoptic Segmentation Forecasting](https://mlsanity.com/paper/2104.03962/) (2021)
- [(Re)2H2O: Autonomous Driving Scenario Generation via Reversely Regularized Hybrid Offline-and-Online Reinforcement Learning](https://mlsanity.com/paper/2302.13726/) (2023)
- [Object-centric Video Prediction without Annotation](https://mlsanity.com/paper/2105.02799/) (2021)
- [WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction](https://mlsanity.com/paper/2211.14308/) (2022)
- [Evaluating Disentanglement of Structured Representations](https://mlsanity.com/paper/2101.04041/) (2021)
- [Joint Forecasting of Panoptic Segmentations with Difference Attention](https://mlsanity.com/paper/2204.07157/) (2022)
- [Self-Supervised Object Detection via Generative Image Synthesis](https://mlsanity.com/paper/2110.09848/) (2021)
- [ObPose: Leveraging Pose for Object-Centric Scene Inference and Generation in 3D](https://mlsanity.com/paper/2206.03591/) (2022)
- [Search for Concepts: Discovering Visual Concepts Using Direct Optimization](https://mlsanity.com/paper/2210.14808/) (2022)
- [ILSGAN: Independent Layer Synthesis for Unsupervised Foreground-Background Segmentation](https://mlsanity.com/paper/2211.13974/) (2022)
- [3D-SSGAN: Lifting 2D Semantics for 3D-Aware Compositional Portrait Synthesis](https://mlsanity.com/paper/2401.03764/) (2024)
- [V3GAN: Decomposing Background, Foreground and Motion for Video Generation](https://mlsanity.com/paper/2203.14074/) (2022)
- [Compositional Scene Modeling with Global Object-Centric Representations](https://mlsanity.com/paper/2211.11500/) (2022)
- [Acceleration method for generating perception failure scenarios based on editing Markov process](https://mlsanity.com/paper/2407.00980/) (2024)

## Abstract (arXiv)

We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained end-to-end on raw, unlabeled data. RELATE combines an object-centric GAN formulation with a model that explicitly accounts for correlations between individual objects. This allows the model to generate realistic scenes and videos from a physically-interpretable parameterization. Furthermore, we show that modeling the object correlation is necessary to learn to disentangle object positions and identity. We find that RELATE is also amenable to physically realistic scene editing and that it significantly outperforms prior art in object-centric scene generation in both synthetic (CLEVR, ShapeStacks) and real-world data (cars). In addition, in contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual fidelity. Source code, datasets and more results are available at http://geometry.cs.ucl.ac.uk/projects/2020/relate/.