# Fast and Robust Hand Tracking Using Detection-Guided Optimization

arXiv: https://arxiv.org/abs/1602.04124 · HTML: https://mlsanity.com/paper/1602.04124/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2016.

## Built on by (most cited first, 30 of 30)

- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [Estimating Egocentric 3D Human Pose in Global Space](https://mlsanity.com/paper/2104.13454/) (2021)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements](https://mlsanity.com/paper/2111.00763/) (2021)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [ZoomNAS: Searching for Whole-body Human Pose Estimation in the Wild](https://mlsanity.com/paper/2208.11547/) (2022)
- [LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer](https://mlsanity.com/paper/2405.07319/) (2024)
- [RelightableHands: Efficient Neural Relighting of Articulated Hand Models](https://mlsanity.com/paper/2302.04866/) (2023)
- [Personal Privacy Protection via Irrelevant Faces Tracking and Pixelation in Video Live Streaming](https://mlsanity.com/paper/2101.01060/) (2021)
- [DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field](https://mlsanity.com/paper/2308.08231/) (2023)
- [3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera](https://mlsanity.com/paper/2312.14157/) (2023)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [EgoRenderer: Rendering Human Avatars from Egocentric Camera Images](https://mlsanity.com/paper/2111.12685/) (2021)
- [MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision](https://mlsanity.com/paper/2310.11696/) (2023)
- [HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild](https://mlsanity.com/paper/2404.13819/) (2024)
- [Stability-driven Contact Reconstruction From Monocular Color Images](https://mlsanity.com/paper/2205.00848/) (2022)
- [HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow](https://mlsanity.com/paper/2210.01692/) (2022)
- [MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild](https://mlsanity.com/paper/2412.13393/) (2024)
- [Hand tracking for clinical applications: validation of the Google MediaPipe Hand (GMH) and the depth-enhanced GMH-D frameworks](https://mlsanity.com/paper/2308.01088/) (2023)
- [D-SCo: Dual-Stream Conditional Diffusion for Monocular Hand-Held Object Reconstruction](https://mlsanity.com/paper/2311.14189/) (2023)
- [FreeCap: Hybrid Calibration-Free Motion Capture in Open Environments](https://mlsanity.com/paper/2411.04469/) (2024)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [At Your Fingertips: Extracting Piano Fingering Instructions from Videos](https://mlsanity.com/paper/2303.03745/) (2023)
- [HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation](https://mlsanity.com/paper/2106.04269/) (2021)
- [Simultaneous prediction of hand gestures, handedness, and hand keypoints using thermal images](https://mlsanity.com/paper/2303.01547/) (2023)
- [Real-Time Hand Gesture Identification in Thermal Images](https://mlsanity.com/paper/2303.02321/) (2023)
- [B'ezier Degradation Modeling for LiDAR-based Human Motion Capture](https://mlsanity.com/paper/2605.19620/) (2026)
- [VEPHand: View-Efficient Photometric Hand Performance Capture at Scale](https://mlsanity.com/paper/2606.15966/) (2026)

## Abstract (arXiv)

Markerless tracking of hands and fingers is a promising enabler for human-computer interaction. However, adoption has been limited because of tracking inaccuracies, incomplete coverage of motions, low framerate, complex camera setups, and high computational requirements. In this paper, we present a fast method for accurately tracking rapid and complex articulations of the hand using a single depth camera. Our algorithm uses a novel detection-guided optimization strategy that increases the robustness and speed of pose estimation. In the detection step, a randomized decision forest classifies pixels into parts of the hand. In the optimization step, a novel objective function combines the detected part labels and a Gaussian mixture representation of the depth to estimate a pose that best fits the depth. Our approach needs comparably less computational resources which makes it extremely fast (50 fps without GPU support). The approach also supports varying static, or moving, camera-to-scene arrangements. We show the benefits of our method by evaluating on public datasets and comparing against previous work.