# Unsupervised Discovery of 3D Physical Objects from Video

arXiv: https://arxiv.org/abs/2007.12348 · HTML: https://mlsanity.com/paper/2007.12348/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2020.

## Built on by (most cited first, 27 of 27)

- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [Object-Centric Slot Diffusion](https://mlsanity.com/paper/2303.10834/) (2023)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Unsupervised Learning of Compositional Energy Concepts](https://mlsanity.com/paper/2111.03042/) (2021)
- [Discovering Objects that Can Move](https://mlsanity.com/paper/2203.10159/) (2022)
- [Learning to Detect Mobile Objects from LiDAR Scans Without Labels](https://mlsanity.com/paper/2203.15882/) (2022)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models](https://mlsanity.com/paper/2306.05357/) (2023)
- [Rotating Features for Object Discovery](https://mlsanity.com/paper/2306.00600/) (2023)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Object Discovery from Motion-Guided Tokens](https://mlsanity.com/paper/2303.15555/) (2023)
- [Neural Groundplans: Persistent Neural Scene Representations from a Single Image](https://mlsanity.com/paper/2207.11232/) (2022)
- [Boosting Object Representation Learning via Motion and Object Continuity](https://mlsanity.com/paper/2211.09771/) (2022)
- [Learning 3D Particle-based Simulators from RGB-D Videos](https://mlsanity.com/paper/2312.05359/) (2023)
- [Learning Discrete Concepts in Latent Hierarchical Models](https://mlsanity.com/paper/2406.00519/) (2024)
- [Unsupervised Segmentation in Real-World Images via Spelke Object Inference](https://mlsanity.com/paper/2205.08515/) (2022)
- [Compositional Image Decomposition with Diffusion Models](https://mlsanity.com/paper/2406.19298/) (2024)
- [Test-time Adaptation with Slot-Centric Models](https://mlsanity.com/paper/2203.11194/) (2022)
- [Neural Language of Thought Models](https://mlsanity.com/paper/2402.01203/) (2024)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning](https://mlsanity.com/paper/2308.14899/) (2023)
- [Perceiving Unseen 3D Objects by Poking the Objects](https://mlsanity.com/paper/2302.13375/) (2023)
- [Binding Dynamics in Rotating Features](https://mlsanity.com/paper/2402.05627/) (2024)
- [Nonparametric Identification of Latent Concepts](https://mlsanity.com/paper/2510.00136/) (2025)
- [Parallelized Spatiotemporal Binding](https://mlsanity.com/paper/2402.17077/) (2024)
- [MOD-UV: Learning Mobile Object Detectors from Unlabeled Videos](https://mlsanity.com/paper/2405.14841/) (2024)

## Abstract (arXiv)

We study the problem of unsupervised physical object discovery. While existing frameworks aim to decompose scenes into 2D segments based off each object's appearance, we explore how physics, especially object interactions, facilitates disentangling of 3D geometry and position of objects from video, in an unsupervised manner. Drawing inspiration from developmental psychology, our Physical Object Discovery Network (POD-Net) uses both multi-scale pixel cues and physical motion cues to accurately segment observable and partially occluded objects of varying sizes, and infer properties of those objects. Our model reliably segments objects on both synthetic and real scenes. The discovered object properties can also be used to reason about physical events.