# SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos

arXiv: https://arxiv.org/abs/2608.16058 · HTML: https://mlsanity.com/paper/2608.16058/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 26 of 26)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks](https://mlsanity.com/paper/2407.12998/) (2024)
- [SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learning](https://mlsanity.com/paper/2505.10251/) (2025)
- [Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot](https://mlsanity.com/paper/2302.09772/) (2023)
- [ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation](https://mlsanity.com/paper/2509.10952/) (2025)
- [Autonomous Robotic Suction to Clear the Surgical Field for Hemostasis using Image-based Blood Flow Detection](https://mlsanity.com/paper/2010.08441/) (2020)
- [Surgical Visual Understanding (SurgVU) Dataset](https://mlsanity.com/paper/2501.09209/) (2025)
- [Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics](https://mlsanity.com/paper/2604.21017/) (2026)
- [SurgRIPE challenge: Benchmark of Surgical Robot Instrument Pose Estimation](https://mlsanity.com/paper/2501.02990/) (2025)
- [Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automation](https://mlsanity.com/paper/2405.18757/) (2024)
- [SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning](https://mlsanity.com/paper/2512.18068/) (2025)
- [Localization and Control of Magnetic Suture Needles in Cluttered Surgical Site with Blood and Tissue](https://mlsanity.com/paper/2105.09481/) (2021)

## Abstract (arXiv)

Learning-based surgical robot autonomy requires large-scale demonstrations with synchronized videos and robot actions, but such data are exceedingly rare in clinical or realistic tissue settings because robot kinematics are typically inaccessible outside controlled research systems. In contrast, phantom data collected on research platforms provide accurate action labels but lack the visual diversity of real tissue. We propose SurgVIL, a framework for scaling surgical robot imitation learning using open-source surgical videos. SurgVIL combines kinematically labeled phantom robot demonstrations with surgical videos from open-source datasets and online sources for policy learning. Since these videos lack robot motion labels, we estimate approximate kinematics as weak supervision. We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting. Across ACT, π0, and GR00T-H backbones, adding surgical videos substantially improves generalization to real-tissue and out-of-distribution settings, suggesting a scalable path from phantom training toward generalizable surgical robot policies.