# Open-Sourced Reinforcement Learning Environments for Surgical Robotics

arXiv: https://arxiv.org/abs/1903.02090 · HTML: https://mlsanity.com/paper/1903.02090/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2019.

## Built on by (most cited first, 26 of 26)

- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Safe Driving via Expert Guided Policy Optimization](https://mlsanity.com/paper/2110.06831/) (2021)
- [Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety](https://mlsanity.com/paper/2105.10682/) (2021)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [Learning from Demonstrations for Autonomous Soft-tissue Retraction](https://mlsanity.com/paper/2110.00336/) (2021)
- [Safe Exploration by Solving Early Terminated MDP](https://mlsanity.com/paper/2107.04200/) (2021)
- [General-purpose foundation models for increased autonomy in robot-assisted surgery](https://mlsanity.com/paper/2401.00678/) (2024)
- [Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learning](https://mlsanity.com/paper/2301.00452/) (2023)
- [Robotic Surgery With Lean Reinforcement Learning](https://mlsanity.com/paper/2105.01006/) (2021)
- [Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario](https://mlsanity.com/paper/2201.00383/) (2022)
- [Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study](https://mlsanity.com/paper/2206.15086/) (2022)
- [Surgical Gym: A high-performance GPU-based platform for reinforcement learning with surgical robots](https://mlsanity.com/paper/2310.04676/) (2023)
- [Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.12953/) (2021)
- [Human-AI Shared Control via Policy Dissection](https://mlsanity.com/paper/2206.00152/) (2022)
- [Hybrid intelligence for dynamic job-shop scheduling with deep reinforcement learning and attention mechanism](https://mlsanity.com/paper/2201.00548/) (2022)
- [FF-SRL: High Performance GPU-Based Surgical Simulation For Robot Learning](https://mlsanity.com/paper/2503.18616/) (2025)
- [Virtual Reality Simulator for Fetoscopic Spina Bifida Repair Surgery](https://mlsanity.com/paper/2208.00169/) (2022)
- [Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systems](https://mlsanity.com/paper/2307.03853/) (2023)
- [Toward a Surgeon-in-the-Loop Ophthalmic Robotic Apprentice using Reinforcement and Imitation Learning](https://mlsanity.com/paper/2311.17693/) (2023)
- [Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning](https://mlsanity.com/paper/2411.14622/) (2024)
- [Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learning](https://mlsanity.com/paper/2502.16328/) (2025)
- [dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scale](https://mlsanity.com/paper/2503.05646/) (2025)
- [TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research](https://mlsanity.com/paper/2511.07412/) (2025)
- [Off-Policy Selection for Initiating Human-Centric Experimental Design](https://mlsanity.com/paper/2410.20017/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) is a machine learning framework for artificially intelligent systems to solve a variety of complex problems. Recent years has seen a surge of successes solving challenging games and smaller domain problems, including simple though non-specific robotic manipulation and grasping tasks. Rapid successes in RL have come in part due to the strong collaborative effort by the RL community to work on common, open-sourced environment simulators such as OpenAI's Gym that allow for expedited development and valid comparisons between different, state-of-art strategies. In this paper, we aim to start the bridge between the RL and the surgical robotics communities by presenting the first open-sourced reinforcement learning environments for surgical robots, called dVRL[3]{dVRL available at https://github.com/ucsdarclab/dVRL}. Through the proposed RL environments, which are functionally equivalent to Gym, we show that it is easy to prototype and implement state-of-art RL algorithms on surgical robotics problems that aim to introduce autonomous robotic precision and accuracy to assisting, collaborative, or repetitive tasks during surgery. Learned policies are furthermore successfully transferable to a real robot. Finally, combining dVRL with the over 40+ international network of da Vinci Surgical Research Kits in active use at academic institutions, we see dVRL as enabling the broad surgical robotics community to fully leverage the newest strategies in reinforcement learning, and for reinforcement learning scientists with no knowledge of surgical robotics to test and develop new algorithms that can solve the real-world, high-impact challenges in autonomous surgery.