# Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction

arXiv: https://arxiv.org/abs/2003.04772 · HTML: https://mlsanity.com/paper/2003.04772/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 16 of 16)

- [Gesture Recognition in Robotic Surgery: a Review](https://mlsanity.com/paper/2102.00027/) (2021)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video](https://mlsanity.com/paper/2109.13593/) (2021)
- [AZTR: Aerial Video Action Recognition with Auto Zoom and Temporal Reasoning](https://mlsanity.com/paper/2303.01589/) (2023)
- [Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario](https://mlsanity.com/paper/2201.00383/) (2022)
- [Runtime Detection of Executional Errors in Robot-Assisted Surgery](https://mlsanity.com/paper/2203.00737/) (2022)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022)
- [Towards Surgical Context Inference and Translation to Gestures](https://mlsanity.com/paper/2302.14237/) (2023)
- [Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment](https://mlsanity.com/paper/2103.04075/) (2021)
- [Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks](https://mlsanity.com/paper/2109.11192/) (2021)
- [Human Gaze Guided Attention for Surgical Activity Recognition](https://mlsanity.com/paper/2203.04752/) (2022)
- [MS-TCRNet: Multi-Stage Temporal Convolutional Recurrent Networks for Action Segmentation Using Sensor-Augmented Kinematics](https://mlsanity.com/paper/2303.07814/) (2023)
- [Is there progress in activity progress prediction?](https://mlsanity.com/paper/2308.05533/) (2023)
- [Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants](https://mlsanity.com/paper/2503.15647/) (2025)
- [Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals](https://mlsanity.com/paper/2603.25906/) (2026)

## Abstract (arXiv)

Surgical gesture recognition is important for surgical data science and computer-aided intervention. Even with robotic kinematic information, automatically segmenting surgical steps presents numerous challenges because surgical demonstrations are characterized by high variability in style, duration and order of actions. In order to extract discriminative features from the kinematic signals and boost recognition accuracy, we propose a multi-task recurrent neural network for simultaneous recognition of surgical gestures and estimation of a novel formulation of surgical task progress. To show the effectiveness of the presented approach, we evaluate its application on the JIGSAWS dataset, that is currently the only publicly available dataset for surgical gesture recognition featuring robot kinematic data. We demonstrate that recognition performance improves in multi-task frameworks with progress estimation without any additional manual labelling and training.