mlsanityData to 14 Oct 2026

Ctrl-World: A Controllable Generative World Model for Robot Manipulation

arXiv 2510.10125 ↗First posted 11 Oct 2025cs.RO cs.AIcited by 167

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Generalist robot policies can now perform a wide range of manipulation skills, but evaluating and improving their ability with unfamiliar objects and instructions remains a significant challenge. Rigorous evaluation requires a large number of real-world rollouts, while systematic improvement demands additional corrective data with expert labels. Both of these processes are slow, costly, and difficult to scale. World models offer a promising, scalable alternative by enabling policies to rollout within imagination space. However, a key challenge is building a controllable world model that can handle multi-step interactions with generalist robot policies. This requires a world model compatible with modern generalist policies by supporting multi-view prediction, fine-grained action control, and consistent long-horizon interactions, which is not achieved by previous works. In this paper, we make a step forward by introducing a controllable multi-view world model that can be used to evaluate and improve the instruction-following ability of generalist robot policies. Our model maintains long-horizon consistency with a pose-conditioned memory retrieval mechanism and achieves precise action control through frame-level action conditioning. Trained on the DROID dataset (95k trajectories, 564 scenes), our model generates spatially and temporally consistent trajectories under novel scenarios and new camera placements for over 20 seconds. We show that our method can accurately rank policy performance without real-world robot rollouts. Moreover, by synthesizing successful trajectories in imagination and using them for supervised fine-tuning, our approach can improve policy success by 44.7%.

Key facts. Cited by 167 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 62 of its 73 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

62 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation29 Jun
  2. 2025WorldEval: World Model as Real-World Robot Policies Evaluator25 May
  3. 2025FLARE: Robot Learning with Implicit World Modeling21 May
  4. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
  5. 2025AdaWorld: Learning Adaptable World Models with Latent Actions24 Mar
  6. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Model11 Dec
  7. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models24 May
  8. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct

Other references · 54

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  9. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  11. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
38 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  3. 2020Mastering Atari with Discrete World Modelscited by 1,007
  4. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  5. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  6. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  7. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  8. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  9. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  10. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  11. 2023Learning Interactive Real-World Simulatorscited by 458
  12. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  13. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  14. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  15. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  16. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  17. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  18. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  19. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  20. 2025Real-Time Execution of Action Chunking Flow Policiescited by 216
  21. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  22. 2025Unified Video Action Modelcited by 205
  23. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  24. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  25. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  26. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  27. 2023Vision-Language Models as Success Detectorscited by 142
  28. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 137
  29. 2025Training Agents Inside of Scalable World Modelscited by 136
  30. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 116
  31. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 102
  32. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 87
  33. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 84
  34. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 82
  35. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  36. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  37. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  38. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 64

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 153.

167 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 142
  3. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  4. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 112
  5. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  6. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 76
  7. 2026Motubrain: An Advanced World Action Model for Robot Controlcited by 57
  8. 2025Evaluating Gemini Robotics Policies in a Veo World Simulatorcited by 51
  9. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 50
  10. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  11. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  12. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 41
38 more
  1. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 40
  2. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 33
  3. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  4. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  5. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 30
  6. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 27
  7. 2026τ0-WM: A Unified Video-Action World Model for Robotic Manipulationcited by 27
  8. 2025NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewardscited by 26
  9. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 22
  10. 2025Act2Goal: From World Model To General Goal-conditioned Policycited by 20
  11. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 18
  12. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
  13. 2025World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertaintycited by 17
  14. 2026FlowWAM: Optical Flow as a Unified Action Representation for World Action Modelscited by 16
  15. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 15
  16. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 15
  17. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  18. 2026Walk through Paintings: Egocentric World Models from Internet Priorscited by 14
  19. 2026EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewardscited by 14
  20. 2026World-Value-Action Model: Implicit Planning for Vision-Language-Action Systemscited by 14
  21. 2026iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Frameworkcited by 14
  22. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 13
  23. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 13
  24. 2026dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Modelcited by 13
  25. 2026Geometry-Aware Implicit Memory for Video World Modelscited by 13
  26. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  27. 2026Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulationcited by 12
  28. 2026Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learningcited by 12
  29. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 12
  30. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 11
  31. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 11
  32. 2026Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesiscited by 10
  33. 2026UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modelingcited by 10
  34. 2026Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Trainingcited by 10
  35. 2026ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Modelscited by 10
  36. 2026WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platformcited by 10
  37. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 10
  38. 2026PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulationcited by 10

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026World Models for Robotic Manipulation: A Survey
  2. 2026DELE-w0.5: Inferring Action from Future Latent State for Robotic Manipulation
  3. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
  4. 2026RM-WAM: Reliable Memory-Augmented World Action Models for Long-Horizon Robot Manipulation
  5. 2026Scaling World Model for Hierarchical Manipulation Policies
  6. 2024Representing Positional Information in Generative World Models for Object Manipulation
  7. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
  8. 2026MemoryWAM: Efficient World Action Modeling with Persistent Memory