mlsanityData to 7 Oct 2026

Gesture Recognition in Robotic Surgery: a Review

arXiv 2102.00027 ↗First posted 29 Jan 2021cs.CV cs.ROcited by 23

Not on the map. Why?

Abstract (arXiv)

Objective: Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. Results: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. Conclusion: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. Significance: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 35 of its 92 references that are in the map. First posted on arXiv 29 Jan 2021.

Builds on

35 of 92 references.

References · 35

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  3. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  4. 2016Temporal Convolutional Networks: A Unified Approach to Action Segmentationcited by 199
  5. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  6. 2019Deep Learning vs. Traditional Computer Visioncited by 94
  7. 2016Segmental Spatiotemporal CNNs for Fine-grained Action Segmentationcited by 78
  8. 2017Multi-Level Discovery of Deep Optionscited by 50
  9. 2019Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Videocited by 36
  10. 2018Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networkscited by 24
  11. 2020Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Searchcited by 23
  12. 2016Recognizing Surgical Activities with Recurrent Neural Networkscited by 21
23 more
  1. 2018Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognitioncited by 21
  2. 2020Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Predictioncited by 16
  3. 2018Deep Reinforcement Learning for Surgical Gesture Segmentation and Classificationcited by 15
  4. 2017TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentationcited by 14
  5. 2020Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sourcescited by 12
  6. 2020Real-Time Context-aware Detection of Unsafe Events in Robot-Assisted Surgerycited by 11
  7. 2020ESAD: Endoscopic Surgeon Action Detection Datasetcited by 11
  8. 2020Symmetric Dilated Convolution for Surgical Gesture Recognitioncited by 11
  9. 2020daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgerycited by 11
  10. 2020Motion2Vec: Semi-Supervised Representation Learning from Surgical Videoscited by 10
  11. 2018End-to-End Fine-Grained Action Segmentation and Recognition Using Conditional Random Field Models and Discriminative Sparse Codingcited by 9
  12. 2020Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networkscited by 8
  13. 2019Surgical Gesture Recognition with Optical Flow onlycited by 7
  14. 2019Automated Surgical Activity Recognition with One Labeled Sequencecited by 7
  15. 2019DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robotscited by 6
  16. 2019Weakly Supervised Recognition of Surgical Gesturescited by 6
  17. 2019More unlabelled data or label more data? A study on semi-supervised laparoscopic image segmentationcited by 5
  18. 2020Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalizationcited by 5
  19. 2020Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgerycited by 5
  20. 2017Progress Estimation and Phase Detection for Sequential Processescited by 4
  21. 2019Multitask Learning of Temporal Connectionism in Convolutional Networks using a Joint Distribution Loss Function to Simultaneously Identify Tools and Phase in Surgical Videoscited by 4
  22. 2018Local Temporal Bilinear Pooling for Fine-grained Action Parsingcited by 2
  23. 2019DASZL: Dynamic Action Signatures for Zero-shot Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 3; 2024: 5; 2025: 6; 2026: 2.

23 papers.

Sort:

By area:

  1. 2022Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgerycited by 10
  2. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  3. 2022Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenariocited by 7
  4. 2024SegSTRONG-C: Segmenting Surgical Tools Robustly On Non-adversarial Generated Corruptions -- An EndoVis'24 Challengecited by 7
  5. 2022Runtime Detection of Executional Errors in Robot-Assisted Surgerycited by 4
  6. 2024Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videoscited by 4
  7. 2023Towards Surgical Context Inference and Translation to Gesturescited by 3
  8. 2024SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgerycited by 3
  9. 2026Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planningcited by 3
  10. 2022Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognitioncited by 2
  11. 2022Quantification of Robotic Surgeries with Vision-Based Deep Learningcited by 1
  12. 2023MS-TCRNet: Multi-Stage Temporal Convolutional Recurrent Networks for Action Segmentation Using Sensor-Augmented Kinematicscited by 1
11 more
  1. 2024Zero-shot Prompt-based Video Encoder for Surgical Gesture Recognitioncited by 1
  2. 2024Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Qualitycited by 1
  3. 2025Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariantscited by 1
  4. 2025M2R2: MultiModal Robotic Representation for Temporal Action Segmentationcited by 1
  5. 2025End to End AI System for Surgical Gesture Sequence Recognition and Clinical Outcome Predictioncited by 1
  6. 2021A Multiple-Instance Learning Approach for the Assessment of Gallbladder Vascularity from Laparoscopic Imagescited by 0
  7. 2021Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperationcited by 0
  8. 2023Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Datacited by 0
  9. 2025Testing Human-Hand Segmentation on In-Distribution and Out-of-Distribution Data in Human-Robot Interactions Using a Deep Ensemble Modelcited by 0
  10. 2025Agentic Surgical AI: Surgeon Style Fingerprinting and Privacy Risk Quantification via Discrete Diffusion in a Vision-Language-Action Frameworkcited by 0
  11. 2026Elastic Spiking Transformers for Efficient Gesture Understandingcited by 0