# ViPro-2: Unsupervised State Estimation via Integrated Dynamics for Guiding Video Prediction

arXiv: https://arxiv.org/abs/2508.06335 · HTML: https://mlsanity.com/paper/2508.06335/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Builds on (most cited first, 21 of 21)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning](https://mlsanity.com/paper/1710.05741/) (2017)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting](https://mlsanity.com/paper/2303.13022/) (2023)
- [Learning Visual Predictive Models of Physics for Playing Billiards](https://mlsanity.com/paper/1511.07404/) (2015)
- [Reasoning About Physical Interactions with Object-Oriented Prediction and Planning](https://mlsanity.com/paper/1812.10972/) (2018)
- [Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video](https://mlsanity.com/paper/1905.11169/) (2019)
- [PDE-Driven Spatiotemporal Disentanglement](https://mlsanity.com/paper/2008.01352/) (2020)
- [Guiding Video Prediction with Explicit Procedural Knowledge](https://mlsanity.com/paper/2406.18220/) (2024)

## Abstract (arXiv)

Predicting future video frames is a challenging task with many downstream applications. Previous work has shown that procedural knowledge enables deep models for complex dynamical settings, however their model ViPro assumed a given ground truth initial symbolic state. We show that this approach led to the model learning a shortcut that does not actually connect the observed environment with the predicted symbolic state, resulting in the inability to estimate states given an observation if previous states are noisy. In this work, we add several improvements to ViPro that enables the model to correctly infer states from observations without providing a full ground truth state in the beginning. We show that this is possible in an unsupervised manner, and extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.