# Runtime Detection of Executional Errors in Robot-Assisted Surgery

arXiv: https://arxiv.org/abs/2203.00737 · HTML: https://mlsanity.com/paper/2203.00737/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2002.08718/) (2020)
- [Gesture Recognition in Robotic Surgery: a Review](https://mlsanity.com/paper/2102.00027/) (2021)
- [Recognizing Surgical Activities with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06329/) (2016)
- [Automated Surgical Skill Assessment in RMIS Training](https://mlsanity.com/paper/1712.08604/) (2017)
- [Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery](https://mlsanity.com/paper/1806.05796/) (2018)
- [Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction](https://mlsanity.com/paper/2003.04772/) (2020)
- [Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources](https://mlsanity.com/paper/2002.02921/) (2020)
- [Real-Time Context-aware Detection of Unsafe Events in Robot-Assisted Surgery](https://mlsanity.com/paper/2005.03611/) (2020)
- [Adverse Events in Robotic Surgery: A Retrospective Study of 14 Years of FDA Data](https://mlsanity.com/paper/1507.03518/) (2015)
- [To Make a Robot Secure: An Experimental Analysis of Cyber Security Threats Against Teleoperated Surgical Robots](https://mlsanity.com/paper/1504.04339/) (2015)
- [Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments](https://mlsanity.com/paper/2106.11962/) (2021)
- [Weakly Supervised Recognition of Surgical Gestures](https://mlsanity.com/paper/1907.10993/) (2019)
- [Context-aware Monitoring in Robotic Surgery](https://mlsanity.com/paper/1901.09802/) (2019)
- [Systems-theoretic Safety Assessment of Robotic Telesurgical Systems](https://mlsanity.com/paper/1504.07135/) (2015)
- [Siamese Neural Networks for EEG-based Brain-computer Interfaces](https://mlsanity.com/paper/2002.00904/) (2020)
- [Continuous Operator Authentication for Teleoperated Systems Using Hidden Markov Models](https://mlsanity.com/paper/2010.14006/) (2020)

## Built on by (most cited first, 4 of 4)

- [Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos](https://mlsanity.com/paper/2406.19217/) (2024)
- [Towards Surgical Context Inference and Translation to Gestures](https://mlsanity.com/paper/2302.14237/) (2023)
- [SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery](https://mlsanity.com/paper/2406.15920/) (2024)
- [Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference](https://mlsanity.com/paper/2308.12789/) (2023)

## Abstract (arXiv)

Despite significant developments in the design of surgical robots and automated techniques for objective evaluation of surgical skills, there are still challenges in ensuring safety in robot-assisted minimally-invasive surgery (RMIS). This paper presents a runtime monitoring system for the detection of executional errors during surgical tasks through the analysis of kinematic data. The proposed system incorporates dual Siamese neural networks and knowledge of surgical context, including surgical tasks and gestures, their distributional similarities, and common error modes, to learn the differences between normal and erroneous surgical trajectories from small training datasets. We evaluate the performance of the error detection using Siamese networks compared to single CNN and LSTM networks trained with different levels of contextual knowledge and training data, using the dry-lab demonstrations of the Suturing and Needle Passing tasks from the JIGSAWS dataset. Our results show that gesture specific task nonspecific Siamese networks obtain micro F1 scores of 0.94 (Siamese-CNN) and 0.95 (Siamese-LSTM), and perform better than single CNN (0.86) and LSTM (0.87) networks. These Siamese networks also outperform gesture nonspecific task specific Siamese-CNN and Siamese-LSTM models for Suturing and Needle Passing.