# SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learning

arXiv: https://arxiv.org/abs/2505.10251 · HTML: https://mlsanity.com/paper/2505.10251/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 34 of its 69 references that are in the map.
- First posted on arXiv 15 May 2025.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [Vision-Based Manipulators Need to Also See from Their Hands](https://mlsanity.com/paper/2203.12677/) (2022)
- [Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2312.10008/) (2023)
- [Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks](https://mlsanity.com/paper/2407.12998/) (2024)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery](https://mlsanity.com/paper/2406.06092/) (2024)
- [Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning](https://mlsanity.com/paper/2011.04813/) (2020)
- [Learning from Demonstrations for Autonomous Soft-tissue Retraction](https://mlsanity.com/paper/2110.00336/) (2021)
- [ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity](https://mlsanity.com/paper/2404.16027/) (2024)
- [General-purpose foundation models for increased autonomy in robot-assisted surgery](https://mlsanity.com/paper/2401.00678/) (2024)
- [LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery](https://mlsanity.com/paper/2302.09606/) (2023)
- [Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control](https://mlsanity.com/paper/1902.01459/) (2019)
- [Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation using Reinforcement Learning](https://mlsanity.com/paper/2306.14085/) (2023)

## Built on by (most cited first, 26 of 26)

- [CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://mlsanity.com/paper/2601.13295/) (2026)
- [Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling](https://mlsanity.com/paper/2512.23162/) (2025)
- [SutureBot: A Precision Framework & Benchmark For Autonomous End-to-End Suturing](https://mlsanity.com/paper/2510.20965/) (2025)
- [Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach](https://mlsanity.com/paper/2509.13774/) (2025)
- [Supervised Mixture-of-Experts for Surgical Grasping and Retraction](https://mlsanity.com/paper/2601.21971/) (2026)
- [Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation](https://mlsanity.com/paper/2607.14852/) (2026)
- [Visuomotor Grasping with World Models for Surgical Robots](https://mlsanity.com/paper/2508.11200/) (2025)
- [Toward Autonomous Soft Robotic Endovascular Navigation via Imitation Learning](https://mlsanity.com/paper/2510.09497/) (2025)
- [Investigating Robot Control Policy Learning for Autonomous X-ray-guided Spine Procedures](https://mlsanity.com/paper/2511.03882/) (2025)
- [Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities](https://mlsanity.com/paper/2603.24318/) (2026)
- [ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration](https://mlsanity.com/paper/2604.04664/) (2026)
- [SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary](https://mlsanity.com/paper/2605.21132/) (2026)
- [Localising under the drape: proprioception in the era of distributed surgical robotic system](https://mlsanity.com/paper/2510.23512/) (2025)
- [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026)
- [Attachment Anchors: A Novel Framework for Laparoscopic Grasping Point Prediction in Colorectal Surgery](https://mlsanity.com/paper/2602.17310/) (2026)
- [Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining](https://mlsanity.com/paper/2602.20500/) (2026)
- [SSP: Safety-guaranteed Surgical Policy via Joint Optimization of Behavioral and Spatial Constraints](https://mlsanity.com/paper/2603.07032/) (2026)
- [Investigating a Policy-Based Formulation for Endoscopic Camera Pose Recovery](https://mlsanity.com/paper/2603.20045/) (2026)
- [AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction](https://mlsanity.com/paper/2604.01371/) (2026)
- [Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following](https://mlsanity.com/paper/2605.28736/) (2026)
- [SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos](https://mlsanity.com/paper/2608.16058/) (2026)
- [DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models](https://mlsanity.com/paper/2609.04555/) (2026)
- [Occupancy Network-Guided Autonomous Robotic Partial Nephrectomy](https://mlsanity.com/paper/2609.16186/) (2026)
- [HumynexSurg-1: A Curated Expert Liposuction Dataset](https://mlsanity.com/paper/2609.23885/) (2026)
- [Multi-modal Representation Fusion for Long-term Microsurgical Anastomosis Trajectory Anticipation](https://mlsanity.com/paper/2610.14492/) (2026)
- [SurgiCLAW: An Interpretable Atomic-Skill Agent for Precision Robotic Micromanipulation](https://mlsanity.com/paper/2610.15766/) (2026)

## Abstract (arXiv)

Research on autonomous surgery has largely focused on simple task automation in controlled environments. However, real-world surgical applications demand dexterous manipulation over extended durations and generalization to the inherent variability of human tissue. These challenges remain difficult to address using existing logic-based or conventional end-to-end learning approaches. To address this gap, we propose a hierarchical framework for performing dexterous, long-horizon surgical steps. Our approach utilizes a high-level policy for task planning and a low-level policy for generating robot trajectories. The high-level planner plans in language space, generating task-level or corrective instructions that guide the robot through the long-horizon steps and correct for the low-level policy's errors. We validate our framework through ex vivo experiments on cholecystectomy, a commonly-practiced minimally invasive procedure, and conduct ablation studies to evaluate key components of the system. Our method achieves a 100% success rate across eight unseen ex vivo gallbladders, operating fully autonomously without human intervention. This work demonstrates step-level autonomy in a surgical procedure, marking a milestone toward clinical deployment of autonomous surgical systems.