# Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model

arXiv: https://arxiv.org/abs/1602.03860 · HTML: https://mlsanity.com/paper/1602.03860/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2016.

## Built on by (most cited first, 15 of 15)

- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement](https://mlsanity.com/paper/2205.07982/) (2022)
- [LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer](https://mlsanity.com/paper/2405.07319/) (2024)
- [Gaussian Shadow Casting for Neural Characters](https://mlsanity.com/paper/2401.06116/) (2024)
- [Stability-driven Contact Reconstruction From Monocular Color Images](https://mlsanity.com/paper/2205.00848/) (2022)
- [Physics-aware Hand-object Interaction Denoising](https://mlsanity.com/paper/2405.11481/) (2024)
- [TriHorn-Net: A Model for Accurate Depth-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2206.07117/) (2022)
- [GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes](https://mlsanity.com/paper/2106.13215/) (2021)
- [Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidance](https://mlsanity.com/paper/2407.12307/) (2024)
- [Lagrangian Hashing for Compressed Neural Field Representations](https://mlsanity.com/paper/2409.05334/) (2024)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [Serial-parallel Multi-Scale Feature Fusion for Anatomy-Oriented Hand Joint Detection](https://mlsanity.com/paper/2102.09757/) (2021)
- [Real-time Recognition of Yoga Poses using computer Vision for Smart Health Care](https://mlsanity.com/paper/2201.07594/) (2022)
- [Projecting Gaussian Ellipsoids While Avoiding Affine Projection Approximation](https://mlsanity.com/paper/2411.07579/) (2024)
- [VEPHand: View-Efficient Photometric Hand Performance Capture at Scale](https://mlsanity.com/paper/2606.15966/) (2026)

## Abstract (arXiv)

Real-time marker-less hand tracking is of increasing importance in human-computer interaction. Robust and accurate tracking of arbitrary hand motion is a challenging problem due to the many degrees of freedom, frequent self-occlusions, fast motions, and uniform skin color. In this paper, we propose a new approach that tracks the full skeleton motion of the hand from multiple RGB cameras in real-time. The main contributions include a new generative tracking method which employs an implicit hand shape representation based on Sum of Anisotropic Gaussians (SAG), and a pose fitting energy that is smooth and analytically differentiable making fast gradient based pose optimization possible. This shape representation, together with a full perspective projection model, enables more accurate hand modeling than a related baseline method from literature. Our method achieves better accuracy than previous methods and runs at 25 fps. We show these improvements both qualitatively and quantitatively on publicly available datasets.