# SurgIRL: Towards Life-Long Learning for Surgical Automation by Incremental Reinforcement Learning

arXiv: https://arxiv.org/abs/2409.15651 · HTML: https://mlsanity.com/paper/2409.15651/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods.
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL for clinical treatment and surgery](https://mlsanity.com/deeprl/rl-for-clinical-treatment-and-surgery/)

## Builds on (most cited first, 29 of 29)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks](https://mlsanity.com/paper/2407.12998/) (2024)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery](https://mlsanity.com/paper/2406.06092/) (2024)
- [Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery](https://mlsanity.com/paper/2109.02323/) (2021)
- [Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot](https://mlsanity.com/paper/2302.09772/) (2023)
- [Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning](https://mlsanity.com/paper/2011.04813/) (2020)
- [Learning from Demonstrations for Autonomous Soft-tissue Retraction](https://mlsanity.com/paper/2110.00336/) (2021)
- [Robotic Tool Tracking under Partially Visible Kinematic Chain: A Unified Approach](https://mlsanity.com/paper/2102.06235/) (2021)
- [Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control](https://mlsanity.com/paper/1902.01459/) (2019)
- [Manipulating Soft Tissues by Deep Reinforcement Learning for Autonomous Robotic Surgery](https://mlsanity.com/paper/1902.05183/) (2019)
- [Autonomous Robotic Suction to Clear the Surgical Field for Hemostasis using Image-based Blood Flow Detection](https://mlsanity.com/paper/2010.08441/) (2020)
- [Robotic Surgery With Lean Reinforcement Learning](https://mlsanity.com/paper/2105.01006/) (2021)
- [Real-to-Sim Deformable Object Manipulation: Optimizing Physics Models with Residual Mappings for Robotic Surgery](https://mlsanity.com/paper/2309.11656/) (2023)
- [Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles](https://mlsanity.com/paper/2112.04071/) (2021)
- [Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation using Reinforcement Learning](https://mlsanity.com/paper/2306.14085/) (2023)
- [Automated pick-up of suturing needles for robotic surgical assistance](https://mlsanity.com/paper/1804.03141/) (2018)
- [Model-Predictive Control of Blood Suction for Surgical Hemostasis using Differentiable Fluid Simulations](https://mlsanity.com/paper/2102.01436/) (2021)
- [Markerless Suture Needle 6D Pose Tracking with Robust Uncertainty Estimation for Autonomous Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2109.12722/) (2021)
- [Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2210.03729/) (2022)
- [STITCH: Augmented Dexterity for Suture Throws Including Thread Coordination and Handoffs](https://mlsanity.com/paper/2404.05151/) (2024)
- [Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2210.11973/) (2022)
- [Automating Vascular Shunt Insertion with the dVRK Surgical Robot](https://mlsanity.com/paper/2211.02293/) (2022)
- [SURESTEP: An Uncertainty-Aware Trajectory Optimization Framework to Enhance Visual Tool Tracking for Robust Surgical Automation](https://mlsanity.com/paper/2404.00123/) (2024)
- [Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocautery](https://mlsanity.com/paper/2404.07185/) (2024)
- [Robotic Constrained Imitation Learning for the Peg Transfer Task in Fundamentals of Laparoscopic Surgery](https://mlsanity.com/paper/2405.03440/) (2024)
- [A Framework For Automated Dissection Along Tissue Boundary](https://mlsanity.com/paper/2310.09669/) (2023)

## Abstract (arXiv)

Surgical automation holds immense potential to improve the outcome and accessibility of surgery. Recent studies use reinforcement learning to learn policies that automate different surgical tasks. However, these policies are developed independently and are limited in their reusability when the task changes, making it more time-consuming when robots learn to solve multiple tasks. Inspired by how human surgeons build their expertise, we train surgical automation policies through Surgical Incremental Reinforcement Learning (SurgIRL). SurgIRL aims to (1) acquire new skills by referring to external policies (knowledge) and (2) accumulate and reuse these skills to solve multiple unseen tasks incrementally (incremental learning). Our SurgIRL framework includes three major components. We first define an expandable knowledge set containing heterogeneous policies that can be helpful for surgical tasks. Then, we propose Knowledge Inclusive Attention Network with mAximum Coverage Exploration (KIAN-ACE), which improves learning efficiency by maximizing the coverage of the knowledge set during the exploration process. Finally, we develop incremental learning pipelines based on KIAN-ACE to accumulate and reuse learned knowledge and solve multiple surgical tasks sequentially. Our simulation experiments show that KIAN-ACE efficiently learns to automate ten surgical tasks separately or incrementally. We also evaluate our learned policies on the da Vinci Research Kit (dVRK) and demonstrate successful sim-to-real transfers.