# Variational Inference for Scalable 3D Object-centric Learning

arXiv: https://arxiv.org/abs/2309.14010 · HTML: https://mlsanity.com/paper/2309.14010/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021)
- [NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2203.11283/) (2022)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)
- [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://mlsanity.com/paper/2104.01148/) (2021)
- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)
- [ROOTS: Object-Centric Representation and Rendering of 3D Scenes](https://mlsanity.com/paper/2006.06130/) (2020)
- [Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views](https://mlsanity.com/paper/2111.07117/) (2021)
- [Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations](https://mlsanity.com/paper/2106.03630/) (2021)
- [Unsupervised object-centric video generation and decomposition in 3D](https://mlsanity.com/paper/2007.06705/) (2020)
- [Unsupervised Discovery and Composition of Object Light Fields](https://mlsanity.com/paper/2205.03923/) (2022)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [Reconstruction Bottlenecks in Object-Centric Generative Models](https://mlsanity.com/paper/2007.06245/) (2020)

## Abstract (arXiv)

We tackle the task of scalable unsupervised object-centric representation learning on 3D scenes. Existing approaches to object-centric representation learning show limitations in generalizing to larger scenes as their learning processes rely on a fixed global coordinate system. In contrast, we propose to learn view-invariant 3D object representations in localized object coordinate systems. To this end, we estimate the object pose and appearance representation separately and explicitly map object representations across views while maintaining object identities. We adopt an amortized variational inference pipeline that can process sequential input and scalably update object latent distributions online. To handle large-scale scenes with a varying number of objects, we further introduce a Cognitive Map that allows the registration and query of objects on a per-scene global map to achieve scalable representation learning. We explore the object-centric neural radiance field (NeRF) as our 3D scene representation, which is jointly modeled within our unsupervised object-centric learning framework. Experimental results on synthetic and real datasets show that our proposed method can infer and maintain object-centric representations of 3D scenes and outperforms previous models.