# Recognizing Surgical Activities with Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1606.06329 · HTML: https://mlsanity.com/paper/1606.06329/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2016.

## Built on by (most cited first, 21 of 21)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2108.11510/) (2021)
- [MIcro-Surgical Anastomose Workflow recognition challenge report](https://mlsanity.com/paper/2103.13111/) (2021)
- [Gesture Recognition in Robotic Surgery: a Review](https://mlsanity.com/paper/2102.00027/) (2021)
- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [A Survey of Video-based Action Quality Assessment](https://mlsanity.com/paper/2204.09271/) (2022)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [Instrument-tissue Interaction Detection Framework for Surgical Video Understanding](https://mlsanity.com/paper/2404.00322/) (2024)
- [Retrieval of surgical phase transitions using reinforcement learning](https://mlsanity.com/paper/2208.00902/) (2022)
- [PEg TRAnsfer Workflow recognition challenge report: Does multi-modal data improve recognition?](https://mlsanity.com/paper/2202.05821/) (2022)
- [Runtime Detection of Executional Errors in Robot-Assisted Surgery](https://mlsanity.com/paper/2203.00737/) (2022)
- [CPR-Coach: Recognizing Composite Error Actions based on Single-class Training](https://mlsanity.com/paper/2309.11718/) (2023)
- [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022)
- [Towards Surgical Context Inference and Translation to Gestures](https://mlsanity.com/paper/2302.14237/) (2023)
- [SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery](https://mlsanity.com/paper/2406.15920/) (2024)
- [Learning Invariant Representation of Tasks for Robust Surgical State Estimation](https://mlsanity.com/paper/2102.09119/) (2021)
- [Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks](https://mlsanity.com/paper/2109.11192/) (2021)
- [Human Gaze Guided Attention for Surgical Activity Recognition](https://mlsanity.com/paper/2203.04752/) (2022)
- [Zero-shot Prompt-based Video Encoder for Surgical Gesture Recognition](https://mlsanity.com/paper/2403.19786/) (2024)
- [Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants](https://mlsanity.com/paper/2503.15647/) (2025)
- [Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data](https://mlsanity.com/paper/2308.02529/) (2023)

## Abstract (arXiv)

We apply recurrent neural networks to the task of recognizing surgical activities from robot kinematics. Prior work in this area focuses on recognizing short, low-level activities, or gestures, and has been based on variants of hidden Markov models and conditional random fields. In contrast, we work on recognizing both gestures and longer, higher-level activites, or maneuvers, and we model the mapping from kinematics to gestures/maneuvers with recurrent neural networks. To our knowledge, we are the first to apply recurrent neural networks to this task. Using a single model and a single set of hyperparameters, we match state-of-the-art performance for gesture recognition and advance state-of-the-art performance for maneuver recognition, in terms of both accuracy and edit distance. Code is available at https://github.com/rdipietro/miccai-2016-surgical-activity-rec .