mlsanityData to 7 Oct 2026

Dreamitate: Real-World Visuomotor Policy Learning via Video Generation

arXiv 2406.16862 ↗First posted 24 Jun 2024cs.RO cs.CVcited by 84

Not on the map. Why?

Cited by 36 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A key challenge in manipulation is learning a policy that can robustly generalize to diverse visual environments. A promising mechanism for learning robust policies is to leverage video generative models, which are pretrained on large-scale datasets of internet videos. In this paper, we propose a visuomotor policy learning framework that fine-tunes a video diffusion model on human demonstrations of a given task. At test time, we generate an example of an execution of the task conditioned on images of a novel scene, and use this synthesized execution directly to control the robot. Our key insight is that using common tools allows us to effortlessly bridge the embodiment gap between the human hand and the robot manipulator. We evaluate our approach on four tasks of increasing complexity and demonstrate that harnessing internet-scale generative models allows the learned policy to achieve a significantly higher degree of generalization than existing behavior cloning approaches.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

29 of 35 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2016Generating Videos with Scene Dynamicscited by 376
  10. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  11. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  12. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
17 more
  1. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  2. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  3. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  4. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  5. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  6. 2017Stochastic Variational Video Predictioncited by 234
  7. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  8. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 198
  9. 2018Stochastic Adversarial Video Predictioncited by 173
  10. 2022Masked World Models for Visual Controlcited by 169
  11. 2024Behavior Generation with Latent Actionscited by 159
  12. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  13. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  14. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  15. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  16. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  17. 2021Learning the Predictability of the Futurecited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 31; 2026: 45.

84 papers.

Sort:

By area:

  1. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  2. 2026World Action Models are Zero-shot Policiescited by 411
  3. 2024Latent Action Pretraining from Videoscited by 302
  4. 2026Causal World Modeling for Robot Controlcited by 289
  5. 2025Unified Video Action Modelcited by 195
  6. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  7. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  8. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  9. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  10. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  11. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  12. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
38 more
  1. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  4. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  5. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 58
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  7. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  8. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  9. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  10. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  11. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  12. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  13. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  14. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  15. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 27
  16. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  17. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  18. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  19. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  20. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 23
  21. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  22. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  23. 2025Robot Learning from a Physical World Modelcited by 18
  24. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 16
  25. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  26. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 14
  27. 2025Solving New Tasks by Adapting Internet Video Knowledgecited by 13
  28. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  29. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 13
  30. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  31. 2025TACO: Taming Diffusion for in-the-wild Video Amodal Completioncited by 10
  32. 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 10
  33. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Playerscited by 10
  34. 2026World Action Models: A Surveycited by 10
  35. 2026Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goalscited by 9
  36. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 9
  37. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learningcited by 8
  38. 2024ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructionscited by 7

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generative Image as Action Models
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills
  3. 2026GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation
  4. 2024Hand-Object Interaction Pretraining from Videos
  5. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
  6. 2025SViP: Sequencing Bimanual Visuomotor Policies with Object-Centric Motion Primitives
  7. 2026GAP: Geometric Anchor Pre-training for Data-Efficient Visuomotor Learning of Manipulation Tasks
  8. 2025Learning Generalizable Visuomotor Policy through Dynamics-Alignment