# Using a single RGB frame for real time 3D hand pose estimation in the wild

arXiv: https://arxiv.org/abs/1712.03866 · HTML: https://mlsanity.com/paper/1712.03866/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2017.

## Built on by (most cited first, 30 of 39)

- [Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation](https://mlsanity.com/paper/2104.14639/) (2021)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [A Skeleton-Driven Neural Occupancy Representation for Articulated Hands](https://mlsanity.com/paper/2109.11399/) (2021)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)
- [Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation](https://mlsanity.com/paper/2107.00434/) (2021)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting](https://mlsanity.com/paper/2307.08243/) (2023)
- [Hand Image Understanding via Deep Multi-Task Learning](https://mlsanity.com/paper/2107.11646/) (2021)
- [Consistent 3D Hand Reconstruction in Video via self-supervised Learning](https://mlsanity.com/paper/2201.09548/) (2022)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field](https://mlsanity.com/paper/2308.08231/) (2023)
- [HandTailor: Towards High-Precision Monocular 3D Hand Recovery](https://mlsanity.com/paper/2102.09244/) (2021)
- [MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision](https://mlsanity.com/paper/2310.11696/) (2023)
- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [Multi-View Video-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2109.11747/) (2021)
- [End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2204.08154/) (2022)
- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [Im2Mesh GAN: Accurate 3D Hand Mesh Recovery from a Single RGB Image](https://mlsanity.com/paper/2101.11239/) (2021)
- [3D Hand Pose and Shape Estimation from RGB Images for Keypoint-Based Hand Gesture Recognition](https://mlsanity.com/paper/2109.13879/) (2021)
- [Dynamic Iterative Refinement for Efficient 3D Hand Pose Estimation](https://mlsanity.com/paper/2111.06500/) (2021)
- [D-SCo: Dual-Stream Conditional Diffusion for Monocular Hand-Held Object Reconstruction](https://mlsanity.com/paper/2311.14189/) (2023)
- [HandyPriors: Physically Consistent Perception of Hand-Object Interactions with Differentiable Priors](https://mlsanity.com/paper/2311.16552/) (2023)
- [HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images](https://mlsanity.com/paper/2603.23997/) (2026)
- [A hybrid classification-regression approach for 3D hand pose estimation using graph convolutional networks](https://mlsanity.com/paper/2105.10902/) (2021)
- [Multi-view Image-based Hand Geometry Refinement using Differentiable Monte Carlo Ray Tracing](https://mlsanity.com/paper/2107.05509/) (2021)
- [Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images](https://mlsanity.com/paper/2307.06038/) (2023)
- [Learning to Score Sign Language with Two-stage Method](https://mlsanity.com/paper/2404.10383/) (2024)

## Abstract (arXiv)

We present a method for the real-time estimation of the full 3D pose of one or more human hands using a single commodity RGB camera. Recent work in the area has displayed impressive progress using RGBD input. However, since the introduction of RGBD sensors, there has been little progress for the case of monocular color input. We capitalize on the latest advancements of deep learning, combining them with the power of generative hand pose estimation techniques to achieve real-time monocular 3D hand pose estimation in unrestricted scenarios. More specifically, given an RGB image and the relevant camera calibration information, we employ a state-of-the-art detector to localize hands. Given a crop of a hand in the image, we run the pretrained network of OpenPose for hands to estimate the 2D location of hand joints. Finally, non-linear least-squares minimization fits a 3D model of the hand to the estimated 2D joint positions, recovering the 3D hand pose. Extensive experimental results provide comparison to the state of the art as well as qualitative assessment of the method in the wild.