mlsanityData to 14 Oct 2026

ViPro-2: Unsupervised State Estimation via Integrated Dynamics for Guiding Video Prediction

arXiv 2508.06335 ↗First posted 8 Aug 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Predicting future video frames is a challenging task with many downstream applications. Previous work has shown that procedural knowledge enables deep models for complex dynamical settings, however their model ViPro assumed a given ground truth initial symbolic state. We show that this approach led to the model learning a shortcut that does not actually connect the observed environment with the predicted symbolic state, resulting in the inability to estimate states given an observation if previous states are noisy. In this work, we add several improvements to ViPro that enables the model to correctly infer states from observations without providing a full ground truth state in the beginning. We show that this is possible in an unsupervised manner, and extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

21 of 25 references.

References · 21

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2020Object-Centric Learning with Slot Attentioncited by 876
  4. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  5. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  6. 2021Conditional Object-Centric Learning from Videocited by 248
  7. 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 205
  8. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 195
  9. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 182
  10. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 166
  11. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  12. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
9 more
  1. 2017A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learningcited by 107
  2. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  3. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  4. 2023ENVIDR: Implicit Differentiable Renderer with Neural Environment Lightingcited by 65
  5. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  6. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  7. 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 45
  8. 2020PDE-Driven Spatiotemporal Disentanglementcited by 19
  9. 2024Guiding Video Prediction with Explicit Procedural Knowledgecited by 1

Built on by

Not cited yet.