# Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery

arXiv: https://arxiv.org/abs/2012.06948 · HTML: https://mlsanity.com/paper/2012.06948/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Video-based fully automatic assessment of open surgery suturing skills](https://mlsanity.com/paper/2110.13972/) (2021)
- [EgoSurgery-Tool: A Dataset of Surgical Tool and Hand Detection from Egocentric Open Surgery Videos](https://mlsanity.com/paper/2406.03095/) (2024)
- [Using Hand Pose Estimation To Automate Open Surgery Training Feedback](https://mlsanity.com/paper/2211.07021/) (2022)
- [Open surgery tool classification and hand utilization using a multi-camera system](https://mlsanity.com/paper/2111.06098/) (2021)
- [EgoSurgery-HTS: A Dataset for Egocentric Hand-Tool Segmentation in Open Surgery Videos](https://mlsanity.com/paper/2503.18755/) (2025)
- [Temporally Guided Articulated Hand Pose Tracking in Surgical Videos](https://mlsanity.com/paper/2101.04281/) (2021)
- [Computer Vision for Increased Operative Efficiency via Identification of Instruments in the Neurosurgical Operating Room: A Proof-of-Concept Study](https://mlsanity.com/paper/2312.03001/) (2023)
- [TSP-OCS: A Time-Series Prediction for Optimal Camera Selection in Multi-Viewpoint Surgical Video Analysis](https://mlsanity.com/paper/2504.06527/) (2025)
- [A Multi-View Pipeline and Benchmark Dataset for 3D Hand Pose Estimation in Surgery](https://mlsanity.com/paper/2601.15918/) (2026)

## Abstract (arXiv)

Open, or non-laparoscopic surgery, represents the vast majority of all operating room procedures, but few tools exist to objectively evaluate these techniques at scale. Current efforts involve human expert-based visual assessment. We leverage advances in computer vision to introduce an automated approach to video analysis of surgical execution. A state-of-the-art convolutional neural network architecture for object detection was used to detect operating hands in open surgery videos. Automated assessment was expanded by combining model predictions with a fast object tracker to enable surgeon-specific hand tracking. To train our model, we used publicly available videos of open surgery from YouTube and annotated these with spatial bounding boxes of operating hands. Our model's spatial detections of operating hands significantly outperforms the detections achieved using pre-existing hand-detection datasets, and allow for insights into intra-operative movement patterns and economy of motion.