# Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery

arXiv: https://arxiv.org/abs/2006.16166 · HTML: https://mlsanity.com/paper/2006.16166/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 30 of 34)

- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [Towards Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2212.04582/) (2022)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms](https://mlsanity.com/paper/2303.13293/) (2023)
- [Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games](https://mlsanity.com/paper/2107.05617/) (2021)
- [Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge](https://mlsanity.com/paper/2102.13644/) (2021)
- [An Empirical Study on Activity Recognition in Long Surgical Videos](https://mlsanity.com/paper/2205.02805/) (2022)
- [Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilities](https://mlsanity.com/paper/2308.03193/) (2023)
- [EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding](https://mlsanity.com/paper/2505.24287/) (2025)
- [DisguisOR: Holistic Face Anonymization for the Operating Room](https://mlsanity.com/paper/2307.14241/) (2023)
- [Review on Action Recognition for Accident Detection in Smart City Transportation Systems](https://mlsanity.com/paper/2208.09588/) (2022)
- [Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room](https://mlsanity.com/paper/2108.11801/) (2021)
- [SurgMAE: Masked Autoencoders for Long Surgical Video Analysis](https://mlsanity.com/paper/2305.11451/) (2023)
- [Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation](https://mlsanity.com/paper/2306.03511/) (2023)
- [Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures](https://mlsanity.com/paper/2106.15309/) (2021)
- [Know your sensORs -- A Modality Study For Surgical Action Classification](https://mlsanity.com/paper/2203.08674/) (2022)
- [Surgical Workflow Recognition: from Analysis of Challenges to Architectural Study](https://mlsanity.com/paper/2203.09230/) (2022)
- [Multi-Modal Unsupervised Pre-Training for Surgical Operating Room Workflow Analysis](https://mlsanity.com/paper/2207.07894/) (2022)
- [Are current long-term video understanding datasets long-term?](https://mlsanity.com/paper/2308.11244/) (2023)
- [ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room](https://mlsanity.com/paper/2312.12250/) (2023)
- [Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room](https://mlsanity.com/paper/2503.13028/) (2025)
- [Adaptation of Surgical Activity Recognition Models Across Operating Rooms](https://mlsanity.com/paper/2207.03083/) (2022)
- [Privacy-Preserving Operating Room Workflow Analysis using Digital Twins](https://mlsanity.com/paper/2504.12552/) (2025)
- [M33D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding](https://mlsanity.com/paper/2309.15313/) (2023)
- [AI-Enhanced Virtual Reality in Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2402.03093/) (2024)
- [Comparison of synthetic dataset generation methods for medical intervention rooms using medical clothing detection as an example](https://mlsanity.com/paper/2209.11493/) (2022)
- [Exploring Self-supervised Skeleton-based Action Recognition in Occluded Environments](https://mlsanity.com/paper/2309.12029/) (2023)
- [Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness](https://mlsanity.com/paper/2407.05448/) (2024)
- [Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition](https://mlsanity.com/paper/2502.13883/) (2025)

## Abstract (arXiv)

Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multi-perspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely annotate the videos with 10 most recognized and clinically relevant classes of activities. Furthermore, we investigate state-of-the-art computer vision action recognition techniques and adapt them for the OR environment and the dataset. First, we fine-tune the Inflated 3D ConvNet (I3D) for clip-level activity recognition on our dataset and use it to extract features from the videos. These features are then fed to a stack of 3 Temporal Gaussian Mixture layers which extracts context from neighboring clips, and eventually go through a Long Short Term Memory network to learn the order of activities in full-length videos. We extensively assess the model and reach a peak performance of 88% mean Average Precision.