mlsanityData to 7 Oct 2026

Deep Visual Foresight for Planning Robot Motion

arXiv 1610.00696 ↗First posted 3 Oct 2016cs.LG cs.AI cs.CVcited by 276

Not on the map. Why?

Cited by 67 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback. Model-based reinforcement learning holds the promise of enabling an agent to learn to predict the effects of its actions, which could provide flexible predictive models for a wide range of tasks and environments, without detailed human supervision. We develop a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data. Our approach does not require a calibrated camera, an instrumented training set-up, nor precise sensing and actuation. Our results show that our method enables a real robot to perform nonprehensile manipulation -- pushing objects -- and can handle novel objects not seen during training.

Key facts. Cited by 276 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 5; 2021: 58; 2022: 40; 2023: 43; 2024: 36; 2025: 40; 2026: 52.

276 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  8. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  9. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  11. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
38 more
  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  2. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  3. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  4. 2024Navigation World Modelscited by 262
  5. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  6. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  8. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  9. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  10. 2022Masked World Models for Visual Controlcited by 169
  11. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  12. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  13. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  14. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  15. 2023Structured World Models from Human Videoscited by 144
  16. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  17. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  18. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  19. 2021Discovering and Achieving Goals via World Modelscited by 131
  20. 2025Unified Vision-Language-Action Modelcited by 128
  21. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  22. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  23. 2025Training Agents Inside of Scalable World Modelscited by 122
  24. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  25. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  26. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  27. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  28. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  29. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  30. 2023On Bringing Robots Homecited by 101
  31. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  32. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  33. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  34. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  35. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 87
  36. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  37. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  38. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83

50 most cited of 276 All 276 (JSON)