mlsanityData to 14 Oct 2026

Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

arXiv 2601.16163 ↗First posted 22 Jan 2026cs.AI cs.ROcited by 300

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Recent video generation models demonstrate remarkable ability to capture complex physical interactions and scene evolution over time. To leverage their spatiotemporal priors, robotics works have adapted video models for policy learning but introduce complexity by requiring multiple stages of post-training and new architectural components for action generation. In this work, we introduce Cosmos Policy, a simple approach for adapting a large pretrained video model (Cosmos-Predict2) into an effective robot policy through a single stage of post-training on the robot demonstration data collected on the target platform, with no architectural modifications. Cosmos Policy learns to directly generate robot actions encoded as latent frames within the video model's latent diffusion process, harnessing the model's pretrained priors and core learning algorithm to capture complex action distributions. Additionally, Cosmos Policy generates future state images and values (expected cumulative rewards), which are similarly encoded as latent frames, enabling test-time planning of action trajectories with higher likelihood of success. In our evaluations, Cosmos Policy achieves state-of-the-art performance on the LIBERO and RoboCasa simulation benchmarks (98.5% and 67.1% average success rates, respectively) and the highest average score in challenging real-world bimanual manipulation tasks, outperforming strong diffusion policies trained from scratch, video model-based policies, and state-of-the-art vision-language-action models fine-tuned on the same robot demonstrations. Furthermore, given policy rollout data, Cosmos Policy can learn from experience to refine its world model and value function and leverage model-based planning to achieve even higher success rates in challenging tasks. We release code, models, and training data at https://research.nvidia.com/labs/dir/cosmos-policy/

Key facts. Cited by 300 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 43 of its 49 references that are in the map. First posted on arXiv 22 Jan 2026.

Builds on

43 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model31 Oct
  2. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation7 Aug
  3. 2025RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation27 Jun
  4. 2025FLARE: Robot Learning with Implicit World Modeling21 May
  5. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy25 Mar
  6. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training22 Feb
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 36

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  11. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
24 more
  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  3. 2023Mastering Diverse Domains through World Modelscited by 1,253
  4. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  5. 2020Mastering Atari with Discrete World Modelscited by 1,007
  6. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  10. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  11. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  13. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  14. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  15. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  16. 2025Unified Video Action Modelcited by 205
  17. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  18. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 160
  19. 2025Video Generators are Robot Policiescited by 92
  20. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  21. 2025HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policycited by 46
  22. 2024A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLMcited by 44
  23. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 17
  24. 2025Latent Policy Steering with Embodiment-Agnostic Pretrained World Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 300.

300 papers.

Sort:

By area:

  1. 2026World Action Models are Zero-shot Policiescited by 465
  2. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  3. 2026Causal World Modeling for Robot Controlcited by 321
  4. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 139
  5. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  6. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 112
  7. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 100
  8. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 76
  9. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  10. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 71
  11. 2026Motubrain: An Advanced World Action Model for Robot Controlcited by 57
  12. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 52
38 more
  1. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 50
  2. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  3. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 50
  4. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  5. 2026AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routingcited by 46
  6. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 45
  7. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 38
  8. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 36
  9. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  10. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  11. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 30
  12. 2026ABot-M0.5: Unified Mobility-and-Manipulation World Action Modelcited by 30
  13. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queriescited by 29
  14. 2026ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignmentcited by 29
  15. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 29
  16. 2026RLDX-1 Technical Reportcited by 28
  17. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 28
  18. 2026τ0-WM: A Unified Video-Action World Model for Robotic Manipulationcited by 27
  19. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 24
  20. 2026VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generationcited by 23
  21. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 20
  22. 2026Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulationcited by 19
  23. 2026RoboTTT: Context Scaling for Robot Policiescited by 19
  24. 2026BPP: Long-Context Robot Imitation Learning by Focusing on Key History Framescited by 18
  25. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 18
  26. 2026Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imaginationcited by 18
  27. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 18
  28. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Actioncited by 18
  29. 2026S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresightcited by 17
  30. 2026WALL-WM: Carving World Action Modeling at the Event Jointscited by 17
  31. 2026Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Modelscited by 17
  32. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Modelscited by 16
  33. 2026FlowWAM: Optical Flow as a Unified Action Representation for World Action Modelscited by 16
  34. 2026Learning 4D Geometric Priors for Inference-Efficient World Action Modelscited by 15
  35. 2026EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewardscited by 14
  36. 2026Gated Memory Policy: In-Context Memorization and Adaptationcited by 14
  37. 2026WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigationcited by 14
  38. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 14

50 most cited of 300 All 300 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
  2. 2026RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations
  3. 2026RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures
  4. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  5. 2026Vid2WAM: Distilling Video Diffusion Priors into World Action Models
  6. 2025Evaluating Gemini Robotics Policies in a Veo World Simulator
  7. 2025MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis
  8. 2026Temporal Policy: History-Initialized Action Generation for Robotic Learning from Demonstration