mlsanityData to 7 Oct 2026

Unsupervised Learning for Physical Interaction through Video Prediction

arXiv 1605.07157 ↗First posted 23 May 2016cs.LG cs.AI cs.CVcited by 256

Not on the map. Why?

Cited by 38 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment. Many existing methods for learning the dynamics of physical interactions require labeled object information. However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical. To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames. Because our model explicitly predicts motion, it is partially invariant to object appearance, enabling it to generalize to previously unseen objects. To explore video prediction for real-world interactive agents, we also introduce a dataset of 59,000 robot interactions involving pushing motions, including a test set with novel objects. In this dataset, accurate prediction of videos conditioned on the robot's future actions amounts to learning a "visual imagination" of different futures based on different courses of action. Our experiments show that our proposed method produces more accurate video predictions both quantitatively and qualitatively, when compared to prior methods.

Key facts. Cited by 256 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 75; 2022: 54; 2023: 45; 2024: 32; 2025: 28; 2026: 22.

256 papers.

Sort:

By area:

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  4. 2024Genie: Generative Interactive Environmentscited by 706
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  6. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  7. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  8. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  9. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  10. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  11. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  12. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
38 more
  1. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  2. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  3. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  4. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  5. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  6. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  7. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  8. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  9. 2023Video Language Planningcited by 160
  10. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  11. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  12. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  13. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  14. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
  15. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  16. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  17. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  18. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  19. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  20. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  21. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 99
  22. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  23. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  24. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  25. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  26. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  27. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  28. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  29. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  30. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  31. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  32. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  33. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  34. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  35. 2021Playable Video Generationcited by 63
  36. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  37. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  38. 2021Predicting Video with VQVAEcited by 61

50 most cited of 256 All 256 (JSON)