Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?
Not on the map. Why?
Abstract (arXiv)
Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream.exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream.exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream.exe will be open-sourced at https://github.com/showlab/Dream.exe.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 40 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
38 of 40 references.
References · 38
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2022Video Diffusion Modelscited by 2,305
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2018World Modelscited by 1,513
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
26 more
- 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
- 2024LTX-Video: Realtime Video Latent Diffusioncited by 581
- 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
- 2023CoTracker: It is Better to Track Togethercited by 522
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
- 2026World Action Models are Zero-shot Policiescited by 465
- 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
- 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
- 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 300
- 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
- 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
- 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
- 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
- 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
- 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
- 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
- 2026Seedance 2.0: Advancing Video Generation for World Complexitycited by 160
- 2025Video Generators are Robot Policiescited by 92
- 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 87
- 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 79
- 2025Kling-Omni Technical Reportcited by 71
- 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64
- 2026MIND: Benchmarking Memory Consistency and Action Control in World Modelscited by 35
- 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
- 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
- 2026DVD: Deterministic Video Depth Estimation with Generative Priorscited by 4
Built on by
Not cited yet.