mlsanityData to 14 Oct 2026

Structured Object-Aware Physics Prediction for Video Modeling and Planning

arXiv 1910.02425 ↗First posted 6 Oct 2019cs.LG cs.CV stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

When humans observe a physical system, they can easily locate objects, understand their interactions, and anticipate future behavior, even in settings with complicated and previously unseen interactions. For computers, however, learning such models from videos in an unsupervised fashion is an unsolved research problem. In this paper, we present STOVE, a novel state-space model for videos, which explicitly reasons about objects and their positions, velocities, and interactions. It is constructed by combining an image model and a dynamics model in compositional manner and improves on previous work by reusing the dynamics model for inference, accelerating and regularizing training. STOVE predicts videos with convincing physical behavior over hundreds of timesteps, outperforms previous unsupervised models, and even approaches the performance of supervised baselines. We further demonstrate the strength of our model as a simulator for sample efficient model-based control in a task with heavily interacting objects.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 13; 2023: 7; 2024: 6; 2025: 4; 2026: 3.

43 papers.

Sort:

By area:

  1. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  2. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  3. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  4. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  5. 2021RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecastingcited by 36
  6. 2022Compositional Scene Representation Learning via Reconstruction: A Surveycited by 27
  7. 2024Entity-Centric Reinforcement Learning for Object Manipulation from Pixelscited by 27
  8. 2021Generative Video Transformer: Can Objects be the Words?cited by 24
  9. 2022Toward Compositional Generalization in Object-Oriented World Modelingcited by 22
  10. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
  11. 2022Neural Implicit Representations for Physical Parameter Inference from a Single Videocited by 19
  12. 2022Unsupervised Multi-object Segmentation by Predicting Probable Motion Patternscited by 19
31 more
  1. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 19
  2. 2023Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learningcited by 15
  3. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  4. 2022Inductive Biases for Object-Centric Representations in the Presence of Complex Texturescited by 13
  5. 2024Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systemscited by 13
  6. 2025Four Principles for Physically Interpretable World Modelscited by 10
  7. 2022Transporters with Visual Foresight for Solving Unseen Rearrangement Taskscited by 9
  8. 2021GATSBI: Generative Agent-centric Spatio-temporal Object Interactioncited by 8
  9. 2021VDSM: Unsupervised Video Disentanglement with State-Space Modeling and Deep Mixtures of Expertscited by 6
  10. 2021Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structurecited by 6
  11. 2022Factored World Models for Zero-Shot Generalization in Robotic Manipulationcited by 6
  12. 2023Granger Causal Interaction Skill Chainscited by 6
  13. 2023Learning Object Permanence from Videos via Latent Imaginationscited by 4
  14. 2025Slot-BERT: Self-supervised Object Discovery in Surgical Videocited by 4
  15. 2022Patch-based Object-centric Transformers for Efficient Video Generationcited by 3
  16. 2021The Impact of Negative Sampling on Contrastive Structured World Modelscited by 2
  17. 2021Self-Supervised Decomposition, Disentanglement and Prediction of Video Sequences while Interpreting Dynamics: A Koopman Perspectivecited by 2
  18. 2022Distilling Governing Laws and Source Input for Dynamical Systems from Videoscited by 2
  19. 2024VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAEcited by 2
  20. 2021Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videoscited by 1
  21. 2022KINet: Unsupervised Forward Models for Robotic Pushing Manipulationcited by 1
  22. 2024Guiding Video Prediction with Explicit Procedural Knowledgecited by 1
  23. 2024Learning Disentangled Representation in Object-Centric Models for Visual Dynamics Prediction via Transformerscited by 1
  24. 2026Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondencecited by 1
  25. 2022Visual Foresight With a Local Dynamics Modelcited by 0
  26. 2023Improving Reliable Navigation under Uncertainty via Predictions Informed by Non-Local Informationcited by 0
  27. 2023Generalization properties of contrastive world modelscited by 0
  28. 2024Graph Neural Networks with Model-based Reinforcement Learning for Multi-agent Systemscited by 0
  29. 2025Compositional Video Synthesis by Temporal Object-Centric Learningcited by 0
  30. 2025Soft Geometric Inductive Bias for Object Centric Dynamicscited by 0
  31. 2026Frozen in a Frame: The Velocity Blind Spot in JEPA World Modelscited by 0