mlsanityData to 7 Oct 2026

Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks

arXiv 2407.12998 ↗First posted 17 Jul 2024cs.ROcited by 38

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

We explore whether surgical manipulation tasks can be learned on the da Vinci robot via imitation learning. However, the da Vinci system presents unique challenges which hinder straight-forward implementation of imitation learning. Notably, its forward kinematics is inconsistent due to imprecise joint measurements, and naively training a policy using such approximate kinematics data often leads to task failure. To overcome this limitation, we introduce a relative action formulation which enables successful policy training and deployment using its approximate kinematics data. A promising outcome of this approach is that the large repository of clinical data, which contains approximate kinematics, may be directly utilized for robot learning without further corrections. We demonstrate our findings through successful execution of three fundamental surgical tasks, including tissue manipulation, needle handling, and knot-tying.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 62 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

42 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects15 Dec
  2. 20223D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery7 Apr

Other references · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  10. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
28 more
  1. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  2. 2019Learning Latent Plans from Playcited by 310
  3. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  4. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  5. 2017One-Shot Imitation Learningcited by 232
  6. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  7. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  8. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  9. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  10. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  11. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  12. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  13. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  14. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  15. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  16. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  17. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  18. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  19. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  20. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  21. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  22. 2019Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Controlcited by 12
  23. 2019Perceptual Values from Observationcited by 11
  24. 2021Robotic Endoscope Control via Autonomous Instrument Trackingcited by 10
  25. 2020Autonomously Navigating a Surgical Tool Inside the Eye by Learning from Demonstrationcited by 8
  26. 2020Towards Autonomous Eye Surgery by Combining Deep Imitation Learning with Optimal Controlcited by 6
  27. 2022Caveats on the first-generation da Vinci Research Kit: latent technical constraints and essential calibrationscited by 5
  28. 2024Robotic Constrained Imitation Learning for the Peg Transfer Task in Fundamentals of Laparoscopic Surgerycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 21; 2026: 13.

38 papers.

Sort:

By area:

  1. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  2. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  3. 2025SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learningcited by 23
  4. 2024TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learningcited by 21
  5. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16
  6. 2025SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Trackingcited by 14
  7. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  8. 2025Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Frameworkcited by 11
  9. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  10. 2025Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modelingcited by 10
  11. 2025Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splattingcited by 5
  12. 2025Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Taskscited by 5
26 more
  1. 2025SutureBot: A Precision Framework & Benchmark For Autonomous End-to-End Suturingcited by 4
  2. 2025EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopycited by 3
  3. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  4. 2026Supervised Mixture-of-Experts for Surgical Grasping and Retractioncited by 2
  5. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  6. 2025dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scalecited by 1
  7. 2025Visuomotor Grasping with World Models for Surgical Robotscited by 1
  8. 2025Investigating Robot Control Policy Learning for Autonomous X-ray-guided Spine Procedurescited by 1
  9. 2026EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigationcited by 1
  10. 2026SurgWMBench: A Vision-Based Benchmark for World-Modeling Surgical Instrument Motion Planningcited by 1
  11. 2024SurgIRL: Towards Life-Long Learning for Surgical Automation by Incremental Reinforcement Learningcited by 0
  12. 2024Learning from Demonstration with Implicit Nonlinear Dynamics Modelscited by 0
  13. 2025Open-Source Multi-Viewpoint Surgical Teleroboticscited by 0
  14. 2025Safe Uncertainty-Aware Learning of Robotic Suturingcited by 0
  15. 2025Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaborationcited by 0
  16. 2025Toward Autonomous Soft Robotic Endovascular Navigation via Imitation Learningcited by 0
  17. 2025Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Predictioncited by 0
  18. 2026On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-trainingcited by 0
  19. 2026Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimizationcited by 0
  20. 2026Investigating a Policy-Based Formulation for Endoscopic Camera Pose Recoverycited by 0
  21. 2026Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Followingcited by 0
  22. 2026SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Roboticscited by 0
  23. 2026Trajectory Divergence Horizon Decision for Reliable Dual-Arm Surgical Subtask Manipulationcited by 0
  24. 2026SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videoscited by 0
  25. 2026DART: Depth-as-Target Pretraining for Surgical Vision Foundation Modelscited by 0
  26. 2026SynIL: Leveraging Synergy for Offline Imitation Learning from Imperfect Demonstration Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning
  2. 2020Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation
  3. 2024SurgicAI: A Hierarchical Platform for Fine-Grained Surgical Policy Learning and Benchmarking
  4. 2026SurgFlow: 3D Object-Centric Contact Flow for Surgical Robot Manipulation ↗
  5. 2024Surgical Task Automation Using Actor-Critic Frameworks and Self-Supervised Imitation Learning
  6. 2025Diffusion Stabilizer Policy for Automated Surgical Robot Manipulations
  7. 2024ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity
  8. 2026Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning