# Physical Object Understanding with a Physically Controllable World Model

arXiv: https://arxiv.org/abs/2606.00439 · HTML: https://mlsanity.com/paper/2606.00439/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 30 May 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control](https://mlsanity.com/paper/2501.03847/) (2025)
- [SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow](https://mlsanity.com/paper/2405.14793/) (2024)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective](https://mlsanity.com/paper/2410.05192/) (2024)
- [Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals](https://mlsanity.com/paper/2505.19386/) (2025)
- [Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation](https://mlsanity.com/paper/2501.05020/) (2025)
- [Unifying (Machine) Vision via Counterfactual World Modeling](https://mlsanity.com/paper/2306.01828/) (2023)
- [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://mlsanity.com/paper/2503.21770/) (2025)
- [Understanding Segment Anything Model: SAM is Biased Towards Texture Rather than Shape](https://mlsanity.com/paper/2311.11465/) (2023)
- [World Modeling with Probabilistic Structure Integration](https://mlsanity.com/paper/2509.09737/) (2025)
- [3D Scene Understanding Through Local Random Access Sequence Modeling](https://mlsanity.com/paper/2504.03875/) (2025)
- [Learning Graphical Models of Images, Videos and Their Spatial Transformations](https://mlsanity.com/paper/1301.3854/) (2013)

## Built on by (most cited first, 1 of 1)

- [MultiEcho: An Experimental Science of Learned Worlds](https://mlsanity.com/paper/2609.33347/) (2026)

## Abstract (arXiv)

A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solving these tasks requires world models capable of inferring distributional states of the world from partial observations - capabilities that current architectures do not provide. We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable, such as appearance and dynamics, conditioned on any other variables. Here, we identify that these models can be trained efficiently with autoregressive sequence modeling, yielding world models from which rich object understanding emerges. First, we demonstrate that our model captures the physical laws governing how objects move by generating multiple plausible future states of the world through sequential inference. Then, by analyzing motion correlations across these futures, we extract objects and articulated object subparts. Having discovered these objects, we show that our world model can manipulate them in 3D. Finally, we demonstrate how physical relationships between objects can be computed from the world model, enabling applications such as Visual Jenga.