# Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot

arXiv: https://arxiv.org/abs/2307.16503 · HTML: https://mlsanity.com/paper/2307.16503/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Divide-and-Conquer Reinforcement Learning](https://mlsanity.com/paper/1711.09874/) (2017)
- [Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization](https://mlsanity.com/paper/2111.07999/) (2021)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery](https://mlsanity.com/paper/2109.02323/) (2021)
- [Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot](https://mlsanity.com/paper/2302.09772/) (2023)
- [Learning from Demonstrations for Autonomous Soft-tissue Retraction](https://mlsanity.com/paper/2110.00336/) (2021)
- [Robot Autonomy for Surgery](https://mlsanity.com/paper/1707.03080/) (2017)
- [Manipulating Soft Tissues by Deep Reinforcement Learning for Autonomous Robotic Surgery](https://mlsanity.com/paper/1902.05183/) (2019)
- [Autonomous task planning and situation awareness in robotic surgery](https://mlsanity.com/paper/2004.08911/) (2020)
- [Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles](https://mlsanity.com/paper/2112.04071/) (2021)
- [A New Tensioning Method using Deep Reinforcement Learning for Surgical Pattern Cutting](https://mlsanity.com/paper/1901.03327/) (2019)
- [Human-Robot Shared Control for Surgical Robot Based on Context-Aware Sim-to-Real Adaptation](https://mlsanity.com/paper/2204.11116/) (2022)
- [Autonomous tissue retraction with a biomechanically informed logic based framework](https://mlsanity.com/paper/2109.02316/) (2021)
- [Training Transition Policies via Distribution Matching for Complex Tasks](https://mlsanity.com/paper/2110.04357/) (2021)

## Built on by (most cited first, 9 of 9)

- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Model](https://mlsanity.com/paper/2509.14138/) (2025)
- [Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automation](https://mlsanity.com/paper/2405.18757/) (2024)
- [SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians](https://mlsanity.com/paper/2405.00956/) (2024)
- [Surgical Task Automation Using Actor-Critic Frameworks and Self-Supervised Imitation Learning](https://mlsanity.com/paper/2409.02724/) (2024)
- [Risk-Sensitive Agent Compositions](https://mlsanity.com/paper/2506.04632/) (2025)
- [SSP: Safety-guaranteed Surgical Policy via Joint Optimization of Behavioral and Spatial Constraints](https://mlsanity.com/paper/2603.07032/) (2026)
- [Regression Test Selection for Updated Capability Modules in Compositional ML Systems via Atomic-Quality Probes](https://mlsanity.com/paper/2604.26689/) (2026)

## Abstract (arXiv)

Reinforcement learning is still struggling with solving long-horizon surgical robot tasks which involve multiple steps over an extended duration of time due to the policy exploration challenge. Recent methods try to tackle this problem by skill chaining, in which the long-horizon task is decomposed into multiple subtasks for easing the exploration burden and subtask policies are temporally connected to complete the whole long-horizon task. However, smoothly connecting all subtask policies is difficult for surgical robot scenarios. Not all states are equally suitable for connecting two adjacent subtasks. An undesired terminate state of the previous subtask would make the current subtask policy unstable and result in a failed execution. In this work, we introduce value-informed skill chaining (ViSkill), a novel reinforcement learning framework for long-horizon surgical robot tasks. The core idea is to distinguish which terminal state is suitable for starting all the following subtask policies. To achieve this target, we introduce a state value function that estimates the expected success probability of the entire task given a state. Based on this value function, a chaining policy is learned to instruct subtask policies to terminate at the state with the highest value so that all subsequent policies are more likely to be connected for accomplishing the task. We demonstrate the effectiveness of our method on three complex surgical robot tasks from SurRoL, a comprehensive surgical simulation platform, achieving high task success rates and execution efficiency. Code is available at hrefhttps://github.com/med-air/ViSkillhttps://github.com/med-air/ViSkill.