# Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

arXiv: https://arxiv.org/abs/2601.16163 · HTML: https://mlsanity.com/paper/2601.16163/

## Key facts

- Cited by 300 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 43 of its 49 references that are in the map.
- First posted on arXiv 22 Jan 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 43)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations](https://mlsanity.com/paper/2412.14803/) (2024)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)

## Built on by (most cited first, 30 of 300)

- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [Fast-WAM: Do World Action Models Need Test-time Future Imagination?](https://mlsanity.com/paper/2603.16666/) (2026)
- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [GigaWorld-Policy: An Efficient Action-Centered World--Action Model](https://mlsanity.com/paper/2603.17240/) (2026)
- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing](https://mlsanity.com/paper/2604.05014/) (2026)
- [Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory](https://mlsanity.com/paper/2604.08995/) (2026)
- [DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control](https://mlsanity.com/paper/2603.10448/) (2026)
- [Being-H0.7: A Latent World-Action Model from Egocentric Videos](https://mlsanity.com/paper/2605.00078/) (2026)
- [Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising](https://mlsanity.com/paper/2604.26694/) (2026)
- [Motubrain: An Advanced World Action Model for Robot Control](https://mlsanity.com/paper/2604.27792/) (2026)
- [LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies](https://mlsanity.com/paper/2606.15768/) (2026)
- [MolmoAct2: Action Reasoning Models for Real-world Deployment](https://mlsanity.com/paper/2605.02881/) (2026)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models](https://mlsanity.com/paper/2606.17846/) (2026)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing](https://mlsanity.com/paper/2606.09811/) (2026)
- [DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation](https://mlsanity.com/paper/2607.05147/) (2026)
- [ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?](https://mlsanity.com/paper/2606.19531/) (2026)
- [Do World Action Models Generalize Better than VLAs? A Robustness Study](https://mlsanity.com/paper/2603.22078/) (2026)
- [Native Video-Action Pretraining for Generalizable Robot Control](https://mlsanity.com/paper/2607.08639/) (2026)
- [Interactive World Simulator for Robot Policy Training and Evaluation](https://mlsanity.com/paper/2603.08546/) (2026)
- [BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generation](https://mlsanity.com/paper/2602.09849/) (2026)
- [ABot-M0.5: Unified Mobility-and-Manipulation World Action Model](https://mlsanity.com/paper/2607.00678/) (2026)
- [ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries](https://mlsanity.com/paper/2603.12942/) (2026)
- [ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment](https://mlsanity.com/paper/2603.23376/) (2026)
- [Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories](https://mlsanity.com/paper/2607.15330/) (2026)
- [RLDX-1 Technical Report](https://mlsanity.com/paper/2605.03269/) (2026)
- [OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation](https://mlsanity.com/paper/2605.06481/) (2026)
- [τ0-WM: A Unified Video-Action World Model for Robotic Manipulation](https://mlsanity.com/paper/2606.01027/) (2026)

## Abstract (arXiv)

Recent video generation models demonstrate remarkable ability to capture complex physical interactions and scene evolution over time. To leverage their spatiotemporal priors, robotics works have adapted video models for policy learning but introduce complexity by requiring multiple stages of post-training and new architectural components for action generation. In this work, we introduce Cosmos Policy, a simple approach for adapting a large pretrained video model (Cosmos-Predict2) into an effective robot policy through a single stage of post-training on the robot demonstration data collected on the target platform, with no architectural modifications. Cosmos Policy learns to directly generate robot actions encoded as latent frames within the video model's latent diffusion process, harnessing the model's pretrained priors and core learning algorithm to capture complex action distributions. Additionally, Cosmos Policy generates future state images and values (expected cumulative rewards), which are similarly encoded as latent frames, enabling test-time planning of action trajectories with higher likelihood of success. In our evaluations, Cosmos Policy achieves state-of-the-art performance on the LIBERO and RoboCasa simulation benchmarks (98.5% and 67.1% average success rates, respectively) and the highest average score in challenging real-world bimanual manipulation tasks, outperforming strong diffusion policies trained from scratch, video model-based policies, and state-of-the-art vision-language-action models fine-tuned on the same robot demonstrations. Furthermore, given policy rollout data, Cosmos Policy can learn from experience to refine its world model and value function and leverage model-based planning to achieve even higher success rates in challenging tasks. We release code, models, and training data at https://research.nvidia.com/labs/dir/cosmos-policy/