mlsanityData to 14 Oct 2026

Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction

arXiv 2003.01460 ↗First posted 3 Mar 2020cs.CVcited by 195

On the map: World models → Latent predictive world models

Abstract (arXiv)

Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video prediction methods. Since physics is too restrictive for describing the full visual content of generic videos, we introduce PhyDNet, a two-branch deep architecture, which explicitly disentangles PDE dynamics from unknown complementary information. A second contribution is to propose a new recurrent physical cell (PhyCell), inspired from data assimilation techniques, for performing PDE-constrained prediction in latent space. Extensive experiments conducted on four various datasets show the ability of PhyDNet to outperform state-of-the-art methods. Ablation studies also highlight the important gain brought out by both disentanglement and PDE-constrained prediction. Finally, we show that PhyDNet presents interesting features for dealing with missing data and long-term forecasting.

Key facts. Cited by 195 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 0 of its 83 references that are in the map. First posted on arXiv 3 Mar 2020.

Builds on

0 of 83 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 30; 2022: 29; 2023: 34; 2024: 39; 2025: 27; 2026: 36.

195 papers.

Sort:

By area:

  1. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  2. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 182
  3. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 152
  4. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  5. 2023PreDiff: Precipitation Nowcasting with Latent Diffusion Modelscited by 98
  6. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  7. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 86
  8. 2021Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularizationcited by 69
  9. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  10. 2022When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learningcited by 65
  11. 2021Encoding physics to learn reaction-diffusion processescited by 61
  12. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 61
38 more
  1. 2021Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modelingcited by 59
  2. 2023Eagle: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformerscited by 57
  3. 2024CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modellingcited by 56
  4. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  5. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 54
  6. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  7. 2021Integrating Expert ODEs into Neural ODEs: Pharmacology and Disease Progressioncited by 48
  8. 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
  9. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  10. 2021Meta-Learning Dynamics Forecasting Using Task Inferencecited by 38
  11. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  12. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 34
  13. 2023SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTMcited by 32
  14. 2022Discovering Nonlinear PDEs from Scarce Data with Physics-encoded Learningcited by 24
  15. 2025Generative Physical AI in Vision: A Surveycited by 24
  16. 2022SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learningcited by 22
  17. 2021Composing Partial Differential Equations with Physics-Aware Neural Networkscited by 21
  18. 2022Learning What and Where: Disentangling Location and Identity Tracking Without Supervisioncited by 21
  19. 2025Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networkscited by 20
  20. 2022Neural Implicit Representations for Physical Parameter Inference from a Single Videocited by 19
  21. 2023Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactionscited by 19
  22. 2023MMVP: Motion-Matrix-based Video Predictioncited by 18
  23. 2022Robust Hybrid Learning With Expert Augmentationcited by 17
  24. 2022MIMO Is All You Need : A Strong Multi-In-Multi-Out Baseline for Video Predictioncited by 17
  25. 2023Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Modelcited by 17
  26. 2024Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcastingcited by 16
  27. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 16
  28. 2022Unsupervised Learning of Equivariant Structure from Sequencescited by 15
  29. 2022Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Spacecited by 14
  30. 2023Implicit Stacked Autoregressive Model for Video Predictioncited by 14
  31. 2022MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networkscited by 13
  32. 2024Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systemscited by 13
  33. 2024E-Motion: Future Motion Simulation via Event Sequence Diffusioncited by 12
  34. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  35. 2024KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filtercited by 11
  36. 2021Deep Time Series Forecasting with Shape and Temporal Criteriacited by 10
  37. 2021A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Predictioncited by 10
  38. 2022Inferring Articulated Rigid Body Dynamics from RGBD Videocited by 10

50 most cited of 195 All 195 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Distilling Physical Priors into Streaming World Models
  2. 2026PhyRPR: Training-Free Physics-Constrained Video Generation
  3. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusion
  4. 2026The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show
  5. 2026Neural Voxel Dynamics: Learning Volumetric Feature Advection for 3D Physics in V-JEPA Latent Space
  6. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suite
  7. 2026StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions
  8. 2025VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models