Guiding Video Prediction with Explicit Procedural Knowledge
Not on the map. Why?
Abstract (arXiv)
We propose a general way to integrate procedural knowledge of a domain into deep learning models. We apply it to the case of video prediction, building on top of object-centric deep models and show that this leads to a better performance than using data-driven models alone. We develop an architecture that facilitates latent space disentanglement in order to use the integrated procedural knowledge, and establish a setup that allows the model to learn the procedural interface in the latent space using the downstream task of video prediction. We contrast the performance to a state-of-the-art data-driven approach and show that problems where purely data-driven approaches struggle can be handled by using knowledge about the domain, providing an alternative to simply collecting more data.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 26 Jun 2024.
Builds on
33 of 45 references.
References · 33
- 2015Spatial Transformer Networkscited by 1,429
- 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
- 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
- 2022Kubric: A scalable dataset generatorcited by 381
- 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
- 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 270
- 2021Conditional Object-Centric Learning from Videocited by 248
- 2020On the Binding Problem in Artificial Neural Networkscited by 233
- 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 205
- 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 195
- 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
- 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
21 more
- 2019Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscited by 117
- 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
- 2017Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledgecited by 96
- 2017Visual Interaction Networkscited by 95
- 2020Augmenting Physical Models with Deep Networks for Complex Dynamics Forecastingcited by 95
- 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
- 2022Continuous PDE Dynamics Forecasting with Implicit Neural Representationscited by 73
- 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
- 2020A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluationcited by 52
- 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
- 2020Learning Physical Graph Representations from Visual Scenescited by 50
- 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 45
- 2021Physics-Guided Deep Learning for Dynamical Systems: A Surveycited by 44
- 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
- 2019Structured Object-Aware Physics Prediction for Video Modeling and Planningcited by 43
- 2022Learning What and Where: Disentangling Location and Identity Tracking Without Supervisioncited by 21
- 2020PDE-Driven Spatiotemporal Disentanglementcited by 19
- 2020Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Surveycited by 16
- 2023Intrinsic Physical Concepts Discovery with Object-Centric Predictive Modelscited by 8
- 2023Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODEcited by 7
- 2022Predicting Physical Object Properties from Videocited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.