# Surgical Gesture Recognition Based on Bidirectional Multi-Layer Independently RNN with Explainable Spatial Feature Extraction

arXiv: https://arxiv.org/abs/2105.00460 · HTML: https://mlsanity.com/paper/2105.00460/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 2 May 2021.

## Builds on (most cited first, 13 of 13)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)
- [Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2002.08718/) (2020)
- [Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification](https://mlsanity.com/paper/1806.08089/) (2018)
- [Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources](https://mlsanity.com/paper/2002.02921/) (2020)
- [End-to-End Fine-Grained Action Segmentation and Recognition Using Conditional Random Field Models and Discriminative Sparse Coding](https://mlsanity.com/paper/1801.09571/) (2018)
- [Weakly Supervised Recognition of Surgical Gestures](https://mlsanity.com/paper/1907.10993/) (2019)
- [Class Activation Map Generation by Representative Class Selection and Multi-Layer Feature Fusion](https://mlsanity.com/paper/1901.07683/) (2019)

## Built on by (most cited first, 5 of 5)

- [Explainable Hierarchical Imitation Learning for Robotic Drink Pouring](https://mlsanity.com/paper/2105.07348/) (2021)
- [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Human Gaze Guided Attention for Surgical Activity Recognition](https://mlsanity.com/paper/2203.04752/) (2022)
- [Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data](https://mlsanity.com/paper/2308.02529/) (2023)

## Abstract (arXiv)

Minimally invasive surgery mainly consists of a series of sub-tasks, which can be decomposed into basic gestures or contexts. As a prerequisite of autonomic operation, surgical gesture recognition can assist motion planning and decision-making, and build up context-aware knowledge to improve the surgical robot control quality. In this work, we aim to develop an effective surgical gesture recognition approach with an explainable feature extraction process. A Bidirectional Multi-Layer independently RNN (BML-indRNN) model is proposed in this paper, while spatial feature extraction is implemented via fine-tuning of a Deep Convolutional Neural Network(DCNN) model constructed based on the VGG architecture. To eliminate the black-box effects of DCNN, Gradient-weighted Class Activation Mapping (Grad-CAM) is employed. It can provide explainable results by showing the regions of the surgical images that have a strong relationship with the surgical gesture classification results. The proposed method was evaluated based on the suturing task with data obtained from the public available JIGSAWS database. Comparative studies were conducted to verify the proposed framework. Results indicated that the testing accuracy for the suturing task based on our proposed method is 87.13%, which outperforms most of the state-of-the-art algorithms.