# Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search

arXiv: https://arxiv.org/abs/2002.08718 · HTML: https://mlsanity.com/paper/2002.08718/

## Key facts

- Cited by 23 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 23 of 23)

- [Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer](https://mlsanity.com/paper/2103.09712/) (2021)
- [LoViT: Long Video Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2305.08989/) (2023)
- [Gesture Recognition in Robotic Surgery: a Review](https://mlsanity.com/paper/2102.00027/) (2021)
- [Accurate Grid Keypoint Learning for Efficient Video Prediction](https://mlsanity.com/paper/2107.13170/) (2021)
- [Future Frame Prediction for Robot-assisted Surgery](https://mlsanity.com/paper/2103.10308/) (2021)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario](https://mlsanity.com/paper/2201.00383/) (2022)
- [Surgical Gesture Recognition Based on Bidirectional Multi-Layer Independently RNN with Explainable Spatial Feature Extraction](https://mlsanity.com/paper/2105.00460/) (2021)
- [Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition](https://mlsanity.com/paper/2302.10834/) (2023)
- [Neural Finite-State Machines for Surgical Phase Recognition](https://mlsanity.com/paper/2411.18018/) (2024)
- [Runtime Detection of Executional Errors in Robot-Assisted Surgery](https://mlsanity.com/paper/2203.00737/) (2022)
- [Neural DNF-MT: A Neuro-symbolic Approach for Learning Interpretable and Editable Policies](https://mlsanity.com/paper/2501.03888/) (2025)
- [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022)
- [DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition](https://mlsanity.com/paper/2409.06217/) (2024)
- [Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment](https://mlsanity.com/paper/2103.04075/) (2021)
- [MoSFormer: Augmenting Temporal Context with Memory of Surgery for Surgical Phase Recognition](https://mlsanity.com/paper/2503.00695/) (2025)
- [Human Gaze Guided Attention for Surgical Activity Recognition](https://mlsanity.com/paper/2203.04752/) (2022)
- [A Non-Anatomical Graph Structure for isolated hand gesture separation in continuous gesture sequences](https://mlsanity.com/paper/2207.07619/) (2022)
- [MS-TCRNet: Multi-Stage Temporal Convolutional Recurrent Networks for Action Segmentation Using Sensor-Augmented Kinematics](https://mlsanity.com/paper/2303.07814/) (2023)
- [Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systems](https://mlsanity.com/paper/2307.03853/) (2023)
- [Toward a Surgeon-in-the-Loop Ophthalmic Robotic Apprentice using Reinforcement and Imitation Learning](https://mlsanity.com/paper/2311.17693/) (2023)
- [CoStoDet-DDPM: Collaborative Training of Stochastic and Deterministic Models Improves Surgical Workflow Anticipation and Recognition](https://mlsanity.com/paper/2503.10216/) (2025)
- [Stabilizing Temporal Inference Dynamics for Online Surgical Phase Recognition](https://mlsanity.com/paper/2605.16387/) (2026)

## Abstract (arXiv)

Automatic surgical gesture recognition is fundamental for improving intelligence in robot-assisted surgery, such as conducting complicated tasks of surgery surveillance and skill evaluation. However, current methods treat each frame individually and produce the outcomes without effective consideration on future information. In this paper, we propose a framework based on reinforcement learning and tree search for joint surgical gesture segmentation and classification. An agent is trained to segment and classify the surgical video in a human-like manner whose direct decisions are re-considered by tree search appropriately. Our proposed tree search algorithm unites the outputs from two designed neural networks, i.e., policy and value network. With the integration of complementary information from distinct models, our framework is able to achieve the better performance than baseline methods using either of the neural networks. For an overall evaluation, our developed approach consistently outperforms the existing methods on the suturing task of JIGSAWS dataset in terms of accuracy, edit score and F1 score. Our study highlights the utilization of tree search to refine actions in reinforcement learning framework for surgical robotic applications.