# Human Gaze Guided Attention for Surgical Activity Recognition

arXiv: https://arxiv.org/abs/2203.04752 · HTML: https://mlsanity.com/paper/2203.04752/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition](https://mlsanity.com/paper/1803.09125/) (2018)
- [LSTA: Long Short-Term Attention for Egocentric Action Recognition](https://mlsanity.com/paper/1811.10698/) (2018)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [DeepPhase: Surgical Phase Recognition in CATARACTS Videos](https://mlsanity.com/paper/1807.10565/) (2018)
- [Integrating Human Gaze into Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/2011.03920/) (2020)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)
- [Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/1807.11794/) (2018)
- [MIcro-Surgical Anastomose Workflow recognition challenge report](https://mlsanity.com/paper/2103.13111/) (2021)
- [Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2002.08718/) (2020)
- [Recognizing Surgical Activities with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06329/) (2016)
- [Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learning](https://mlsanity.com/paper/1806.00466/) (2018)
- [Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks](https://mlsanity.com/paper/1908.07319/) (2019)
- [Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction](https://mlsanity.com/paper/2003.04772/) (2020)
- [Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification](https://mlsanity.com/paper/1806.08089/) (2018)
- [Symmetric Dilated Convolution for Surgical Gesture Recognition](https://mlsanity.com/paper/2007.06373/) (2020)
- [Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networks](https://mlsanity.com/paper/2001.03728/) (2020)
- [Joint Surgical Gesture and Task Classification with Multi-Task and Multimodal Learning](https://mlsanity.com/paper/1805.00721/) (2018)
- [Surgical Gesture Recognition Based on Bidirectional Multi-Layer Independently RNN with Explainable Spatial Feature Extraction](https://mlsanity.com/paper/2105.00460/) (2021)

## Built on by (most cited first, 1 of 1)

- [Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding](https://mlsanity.com/paper/2510.21356/) (2025)

## Abstract (arXiv)

Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attention mechanism for activity recognition in surgical videos. Our model consists of an I3D-based architecture, learns spatio-temporal features using 3D convolutions, as well as learns an attention map using human gaze as supervision. We evaluate our model on the Suturing task of JIGSAWS which is a publicly available surgical video understanding dataset. To our knowledge, we are the first to use human gaze for surgical activity recognition. Our results and ablation studies support the contribution of using human gaze to guide attention by outperforming state-of-the art models with an accuracy of 85.4%.