# Using Hand Pose Estimation To Automate Open Surgery Training Feedback

arXiv: https://arxiv.org/abs/2211.07021 · HTML: https://mlsanity.com/paper/2211.07021/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 13 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [Towards Unified Surgical Skill Assessment](https://mlsanity.com/paper/2106.01035/) (2021)
- [Evaluating surgical skills from kinematic data using convolutional neural networks](https://mlsanity.com/paper/1806.02750/) (2018)
- [MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/2006.09220/) (2020)
- [Video-based fully automatic assessment of open surgery suturing skills](https://mlsanity.com/paper/2110.13972/) (2021)
- [A real-time spatiotemporal AI model analyzes skill in open surgical videos](https://mlsanity.com/paper/2112.07219/) (2021)
- [Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery](https://mlsanity.com/paper/2012.06948/) (2020)
- [Surgical Gesture Recognition with Optical Flow only](https://mlsanity.com/paper/1904.01143/) (2019)
- [Open surgery tool classification and hand utilization using a multi-camera system](https://mlsanity.com/paper/2111.06098/) (2021)
- [Gesture Recognition in RGB Videos UsingHuman Body Keypoints and Dynamic Time Warping](https://mlsanity.com/paper/1906.12171/) (2019)
- [Temporally Guided Articulated Hand Pose Tracking in Surgical Videos](https://mlsanity.com/paper/2101.04281/) (2021)

## Built on by (most cited first, 6 of 6)

- [Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Camera](https://mlsanity.com/paper/2401.10037/) (2024)
- [Monocular pose estimation of articulated open surgery tools -- in the wild](https://mlsanity.com/paper/2407.12138/) (2024)
- [Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation](https://mlsanity.com/paper/2508.20830/) (2025)
- [Explainable AI for Automated User-specific Feedback in Surgical Skill Acquisition](https://mlsanity.com/paper/2508.02593/) (2025)
- [A Multi-View Pipeline and Benchmark Dataset for 3D Hand Pose Estimation in Surgery](https://mlsanity.com/paper/2601.15918/) (2026)
- [ExpOS: Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction](https://mlsanity.com/paper/2605.23653/) (2026)

## Abstract (arXiv)

Purpose: This research aims to facilitate the use of state-of-the-art computer vision algorithms for the automated training of surgeons and the analysis of surgical footage. By estimating 2D hand poses, we model the movement of the practitioner's hands, and their interaction with surgical instruments, to study their potential benefit for surgical training. Methods: We leverage pre-trained models on a publicly-available hands dataset to create our own in-house dataset of 100 open surgery simulation videos with 2D hand poses. We also assess the ability of pose estimations to segment surgical videos into gestures and tool-usage segments and compare them to kinematic sensors and I3D features. Furthermore, we introduce 6 novel surgical dexterity proxies stemming from domain experts' training advice, all of which our framework can automatically detect given raw video footage. Results: State-of-the-art gesture segmentation accuracy of 88.35% on the Open Surgery Simulation dataset is achieved with the fusion of 2D poses and I3D features from multiple angles. The introduced surgical skill proxies presented significant differences for novices compared to experts and produced actionable feedback for improvement. Conclusion: This research demonstrates the benefit of pose estimations for open surgery by analyzing their effectiveness in gesture segmentation and skill assessment. Gesture segmentation using pose estimations achieved comparable results to physical sensors while being remote and markerless. Surgical dexterity proxies that rely on pose estimation proved they can be used to work towards automated training feedback. We hope our findings encourage additional collaboration on novel skill proxies to make surgical training more efficient.