# Temporally Guided Articulated Hand Pose Tracking in Surgical Videos

arXiv: https://arxiv.org/abs/2101.04281 · HTML: https://mlsanity.com/paper/2101.04281/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 12 Jan 2021.

## Builds on (most cited first, 24 of 24)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [PoseTrack: A Benchmark for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1710.10000/) (2017)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Combining detection and tracking for human pose estimation in videos](https://mlsanity.com/paper/2003.13743/) (2020)
- [Learning Temporal Pose Estimation from Sparsely-Labeled Videos](https://mlsanity.com/paper/1906.04016/) (2019)
- [Multi-person Articulated Tracking with Spatial and Temporal Embeddings](https://mlsanity.com/paper/1903.09214/) (2019)
- [3D Hand Pose Tracking and Estimation Using Stereo Matching](https://mlsanity.com/paper/1610.07214/) (2016)
- [Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos](https://mlsanity.com/paper/1812.01366/) (2018)
- [Efficient Online Multi-Person 2D Pose Tracking with Recurrent Spatio-Temporal Affinity Fields](https://mlsanity.com/paper/1811.11975/) (2018)
- [Concurrent Segmentation and Localization for Tracking of Surgical Instruments](https://mlsanity.com/paper/1703.10701/) (2017)
- [Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection](https://mlsanity.com/paper/2008.00936/) (2020)
- [Large-scale Multiview 3D Hand Pose Dataset](https://mlsanity.com/paper/1707.03742/) (2017)
- [LightTrack: A Generic Framework for Online Top-Down Human Pose Tracking](https://mlsanity.com/paper/1905.02822/) (2019)
- [RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network](https://mlsanity.com/paper/1905.08663/) (2019)
- [Attention! A Lightweight 2D Hand Pose Estimation Approach](https://mlsanity.com/paper/2001.08047/) (2020)
- [Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery](https://mlsanity.com/paper/2012.06948/) (2020)

## Built on by (most cited first, 2 of 2)

- [Using Hand Pose Estimation To Automate Open Surgery Training Feedback](https://mlsanity.com/paper/2211.07021/) (2022)
- [MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives](https://mlsanity.com/paper/2501.04184/) (2025)

## Abstract (arXiv)

Articulated hand pose tracking is an under-explored problem that carries the potential for use in an extensive number of applications, especially in the medical domain. With a robust and accurate tracking system on surgical videos, the motion dynamics and movement patterns of the hands can be captured and analyzed for many rich tasks. In this work, we propose a novel hand pose estimation model, CondPose, which improves detection and tracking accuracy by incorporating a pose prior into its prediction. We show improvements over state-of-the-art methods which provide frame-wise independent predictions, by following a temporally guided approach that effectively leverages past predictions. We collect Surgical Hands, the first dataset that provides multi-instance articulated hand pose annotations for videos. Our dataset provides over 8.1k annotated hand poses from publicly available surgical videos and bounding boxes, pose annotations, and tracking IDs to enable multi-instance tracking. When evaluated on Surgical Hands, we show our method outperforms the state-of-the-art approach using mean Average Precision (mAP), to measure pose estimation accuracy, and Multiple Object Tracking Accuracy (MOTA), to assess pose tracking performance. In comparison to a frame-wise independent strategy, we show greater performance in detecting and tracking hand poses and more substantial impact on localization accuracy. This has positive implications in generating more accurate representations of hands in the scene to be used for targeted downstream tasks.