mlsanityData to 7 Oct 2026

Unsupervised Learning of Object Structure and Dynamics from Videos

arXiv 1906.07889 ↗First posted 19 Jun 2019cs.CVcited by 92

Not on the map. Why?

Abstract (arXiv)

Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stochastic dynamics model of the keypoints. Future frames are reconstructed from the keypoints and a reference frame. By modeling dynamics in the keypoint coordinate space, we achieve stable learning and avoid compounding of errors in pixel space. Our method improves upon unstructured representations both for pixel-level video prediction and for downstream tasks requiring object-level understanding of motion dynamics. We evaluate our model on diverse datasets: a multi-agent sports dataset, the Human3.6M dataset, and datasets based on continuous control tasks from the DeepMind Control Suite. The spatially structured representation outperforms unstructured representations on a range of motion-related tasks such as object tracking, action recognition and reward prediction.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 29; 2022: 27; 2023: 14; 2024: 11; 2025: 5; 2026: 5.

92 papers.

Sort:

By area:

  1. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  2. 2022Masked World Models for Visual Controlcited by 169
  3. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  5. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  6. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  7. 2021Region Similarity Representation Learningcited by 96
  8. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  9. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  10. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  11. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  12. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
38 more
  1. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  2. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  3. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  4. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  5. 2021Controllable Animation of Fluid Elements in Still Imagescited by 44
  6. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  7. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  8. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  9. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  10. 2021Video Autoencoder: self-supervised disentanglement of static 3D structure and motioncited by 34
  11. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  12. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  13. 2021Dynamic Neural Garmentscited by 27
  14. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  15. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  16. 2022OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Cloudscited by 24
  17. 2021Revisiting Hierarchical Approach for Persistent Long-Term Video Predictioncited by 23
  18. 2021Self-Supervised Keypoint Discovery in Behavioral Videoscited by 23
  19. 2023Dynamic-Resolution Model Learning for Object Pile Manipulationcited by 22
  20. 2023Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videoscited by 21
  21. 20233D Implicit Transporter for Temporally Consistent Keypoint Discoverycited by 20
  22. 2021GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generationcited by 19
  23. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  24. 2023Unifying (Machine) Vision via Counterfactual World Modelingcited by 18
  25. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  26. 2022AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypointscited by 17
  27. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  28. 2024Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Modelscited by 16
  29. 2022Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Spacecited by 14
  30. 2021Accurate Grid Keypoint Learning for Efficient Video Predictioncited by 13
  31. 2023MOSO: Decomposing MOtion, Scene and Object for Video Predictioncited by 12
  32. 2021Future Frame Prediction for Robot-assisted Surgerycited by 11
  33. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  34. 2022MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Posecited by 10
  35. 2021Conditional Temporal Variational AutoEncoder for Action Video Predictioncited by 9
  36. 2022Transporters with Visual Foresight for Solving Unseen Rearrangement Taskscited by 9
  37. 2021GATSBI: Generative Agent-centric Spatio-temporal Object Interactioncited by 8
  38. 2022Stochastic Video Prediction with Structure and Motioncited by 8

50 most cited of 92 All 92 (JSON)