# Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

arXiv: https://arxiv.org/abs/2512.23162 · HTML: https://mlsanity.com/paper/2512.23162/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Medical and surgical world models, one of 11 directions in World models.
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 29 Dec 2025.

## Placed in

- World models / [Medical and surgical world models](https://mlsanity.com/wm/medical-and-surgical-world-models/)
- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 30 of 37)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation](https://mlsanity.com/paper/2412.15109/) (2024)
- [Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2409.16283/) (2024)
- [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Endora: Video Generation Models as Endoscopy Simulators](https://mlsanity.com/paper/2403.11050/) (2024)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room](https://mlsanity.com/paper/2005.03501/) (2020)
- [Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks](https://mlsanity.com/paper/2407.12998/) (2024)
- [SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence](https://mlsanity.com/paper/2506.02555/) (2025)
- [SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learning](https://mlsanity.com/paper/2505.10251/) (2025)
- [SurGen: Text-Guided Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2408.14028/) (2024)

## Built on by (most cited first, 11 of 11)

- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026)
- [Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling](https://mlsanity.com/paper/2607.27511/) (2026)
- [SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence](https://mlsanity.com/paper/2603.16822/) (2026)
- [Towards World Models in Biomedical Research](https://mlsanity.com/paper/2606.05925/) (2026)
- [SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation](https://mlsanity.com/paper/2605.16530/) (2026)
- [SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planning](https://mlsanity.com/paper/2608.08070/) (2026)
- [CrossScope: A Role-Asymmetric World Model for Joint Dual-Scope Surgical Video Prediction](https://mlsanity.com/paper/2608.03211/) (2026)
- [Surg-UniWorld: A Unified Surgical World Model with Multimodal Control Experts](https://mlsanity.com/paper/2608.06770/) (2026)
- [Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning](https://mlsanity.com/paper/2608.20284/) (2026)
- [Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation](https://mlsanity.com/paper/2610.09800/) (2026)

## Abstract (arXiv)

Data scarcity remains a fundamental barrier to achieving fully autonomous surgical robots. While large scale vision language action (VLA) models have shown impressive generalization in household and industrial manipulation by leveraging paired video action data from diverse domains, surgical robotics suffers from the paucity of datasets that include both visual observations and accurate robot kinematics. In contrast, vast corpora of surgical videos exist, but they lack corresponding action labels, preventing direct application of imitation learning or VLA training. In this work, we aim to alleviate this problem by learning policy models from Cosmos-H-Surgical, a world model designed for surgical physical AI. We curated the Surgical Action Text Alignment (SATA) dataset with detailed action description specifically for surgical robots. Then we built Cosmos-H-Surgical based on the most advanced physical AI world model and SATA. It's able to generate diverse, generalizable and realistic surgery videos. We are also the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos, producing synthetic paired video action data. We demonstrate that a surgical VLA policy trained with these augmented data significantly outperforms models trained only on real demonstrations on a real surgical robot platform. Our approach offers a scalable path toward autonomous surgical skill acquisition by leveraging the abundance of unlabeled surgical video and generative world modeling, thus opening the door to generalizable and data efficient surgical robot policies.