# Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learning

arXiv: https://arxiv.org/abs/1806.00466 · HTML: https://mlsanity.com/paper/1806.00466/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2018.

## Built on by (most cited first, 18 of 18)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge](https://mlsanity.com/paper/2102.13644/) (2021)
- [One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video](https://mlsanity.com/paper/2103.12988/) (2021)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Objective Surgical Skills Assessment and Tool Localization: Results from the MICCAI 2021 SimSurgSkill Challenge](https://mlsanity.com/paper/2212.04448/) (2022)
- [Spatiotemporal Deformable Scene Graphs for Complex Activity Detection](https://mlsanity.com/paper/2104.08194/) (2021)
- [Adaptation of Surgical Activity Recognition Models Across Operating Rooms](https://mlsanity.com/paper/2207.03083/) (2022)
- [SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition](https://mlsanity.com/paper/2306.08859/) (2023)
- [Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery](https://mlsanity.com/paper/2402.01974/) (2024)
- [Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis](https://mlsanity.com/paper/2501.00525/) (2024)
- [Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasets](https://mlsanity.com/paper/2407.19714/) (2024)
- [Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI](https://mlsanity.com/paper/2504.19918/) (2025)
- [Human Gaze Guided Attention for Surgical Activity Recognition](https://mlsanity.com/paper/2203.04752/) (2022)
- [Quantification of Robotic Surgeries with Vision-Based Deep Learning](https://mlsanity.com/paper/2205.03028/) (2022)
- [Rethinking Generalization Performance of Surgical Phase Recognition with Expert-Generated Annotations](https://mlsanity.com/paper/2110.11626/) (2021)
- [Automatic Detection of Out-of-body Frames in Surgical Videos for Privacy Protection Using Self-supervised Learning and Minimal Labels](https://mlsanity.com/paper/2303.18106/) (2023)
- [SurGNN: Explainable visual scene understanding and assessment of surgical skill using graph neural networks](https://mlsanity.com/paper/2308.13073/) (2023)

## Abstract (arXiv)

Adverse surgical outcomes are costly to patients and hospitals. Approaches to benchmark surgical care are often limited to gross measures across the entire procedure despite the performance of particular tasks being largely responsible for undesirable outcomes. In order to produce metrics from tasks as opposed to the whole procedure, methods to recognize automatically individual surgical tasks are needed. In this paper, we propose several approaches to recognize surgical activities in robot-assisted minimally invasive surgery using deep learning. We collected a clinical dataset of 100 robot-assisted radical prostatectomies (RARP) with 12 tasks each and propose `RP-Net', a modified version of InceptionV3 model, for image based surgical activity recognition. We achieve an average precision of 80.9% and average recall of 76.7% across all tasks using RP-Net which out-performs all other RNN and CNN based models explored in this paper. Our results suggest that automatic surgical activity recognition during RARP is feasible and can be the foundation for advanced analytics.