mlsanityData to 7 Oct 2026

Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario

arXiv 2201.00383 ↗First posted 2 Jan 2022cs.RO cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Robot-assisted surgery has become progressively more and more popular due to its clinical advantages. In the meanwhile, the artificial intelligence and augmented reality in robotic surgery are developing rapidly and receive lots of attention. However, current methods have not discussed the coherent integration of AI and AR in robotic surgery. In this paper, we develop a novel system by seamlessly merging artificial intelligence module and augmented reality visualization to automatically generate the surgical guidance for robotic surgery education. Specifically, we first leverage reinforcement leaning to learn from expert demonstration and then generate 3D guidance trajectory, providing prior context information of the surgical procedure. Along with other information such as text hint, the 3D trajectory is then overlaid in the stereo view of dVRK, where the user can perceive the 3D guidance and learn the procedure. The proposed system is evaluated through a preliminary experiment on surgical education task peg-transfer, which proves its feasibility and potential as the next generation of robot-assisted surgery education solution.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 82 references that are in the map. First posted on arXiv 2 Jan 2022.

Builds on

22 of 82 references.

References · 22

  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  3. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  4. 2021Temporal Memory Relation Network for Workflow Recognition from Surgical Videocited by 66
  5. 2021E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perceptioncited by 46
  6. 2020Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Videocited by 35
  7. 2021SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learningcited by 28
  8. 2021MIcro-Surgical Anastomose Workflow recognition challenge reportcited by 27
  9. 2019Open-Sourced Reinforcement Learning Environments for Surgical Roboticscited by 24
  10. 2019Lane Change Decision-making through Deep Reinforcement Learning with Rule-based Constraintscited by 24
  11. 2020Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Searchcited by 23
  12. 2021Gesture Recognition in Robotic Surgery: a Reviewcited by 23
10 more
  1. 2020Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planningcited by 19
  2. 2020Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgerycited by 17
  3. 2020Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Predictioncited by 16
  4. 2018Deep Reinforcement Learning for Surgical Gesture Segmentation and Classificationcited by 15
  5. 2019Solving the Robot-World Hand-Eye(s) Calibration Problem with Iterative Methodscited by 9
  6. 2019A New Tensioning Method using Deep Reinforcement Learning for Surgical Pattern Cuttingcited by 8
  7. 2017Real-time Geometry-Aware Augmented Reality in Minimally Invasive Surgerycited by 3
  8. 2020A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noisescited by 3
  9. 2017A surgical system for automatic registration, stiffness mapping and dynamic image overlaycited by 1
  10. 2020A Reliable Gravity Compensation Control Strategy for dVRK Robotic Arms With Nonlinear Disturbance Forcescited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 2.

7 papers.

  1. 2023Efficient Deformable Tissue Reconstruction via Orthogonal Neural Planecited by 20
  2. 2023Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learningcited by 11
  3. 20223D endoscopic depth estimation using 3D surface-aware constraintscited by 3
  4. 2024FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videoscited by 3
  5. 2022Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interactioncited by 2
  6. 2025dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scalecited by 1
  7. 2025Setup-Invariant Augmented Reality for Teaching by Demonstration with Surgical Robotscited by 1