Human Gaze Guided Attention for Surgical Activity Recognition
Not on the map. Why?
Abstract (arXiv)
Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attention mechanism for activity recognition in surgical videos. Our model consists of an I3D-based architecture, learns spatio-temporal features using 3D convolutions, as well as learns an attention map using human gaze as supervision. We evaluate our model on the Suturing task of JIGSAWS which is a publicly available surgical video understanding dataset. To our knowledge, we are the first to use human gaze for surgical activity recognition. Our results and ablation studies support the contribution of using human gaze to guide attention by outperforming state-of-the art models with an accuracy of 85.4%.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 9 Mar 2022.
Builds on
22 of 39 references.
References · 22
- 2017Attention Is All You Needcited by 59,932
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2017The Kinetics Human Action Video Datasetcited by 1,894
- 2020TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networkscited by 139
- 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
- 2018LSTA: Long Short-Term Attention for Egocentric Action Recognitioncited by 57
- 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
- 2018DeepPhase: Surgical Phase Recognition in CATARACTS Videoscited by 47
- 2020Integrating Human Gaze into Attention for Egocentric Activity Recognitioncited by 42
- 2019Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Videocited by 36
- 2018Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognitioncited by 28
- 2021MIcro-Surgical Anastomose Workflow recognition challenge reportcited by 27
10 more
- 2020Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Searchcited by 23
- 2016Recognizing Surgical Activities with Recurrent Neural Networkscited by 21
- 2018Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learningcited by 18
- 2019Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networkscited by 18
- 2020Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Predictioncited by 16
- 2018Deep Reinforcement Learning for Surgical Gesture Segmentation and Classificationcited by 15
- 2020Symmetric Dilated Convolution for Surgical Gesture Recognitioncited by 11
- 2020Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networkscited by 8
- 2018Joint Surgical Gesture and Task Classification with Multi-Task and Multimodal Learningcited by 5
- 2021Surgical Gesture Recognition Based on Bidirectional Multi-Layer Independently RNN with Explainable Spatial Feature Extractioncited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.