# 3D Hand Shape and Pose from Images in the Wild

arXiv: https://arxiv.org/abs/1902.03451 · HTML: https://mlsanity.com/paper/1902.03451/

## Key facts

- Cited by 190 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2019.

## Built on by (most cited first, 30 of 190)

- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021)
- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [Interacting Attention Graph for Single Image Two-Hand Reconstruction](https://mlsanity.com/paper/2203.09364/) (2022)
- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis](https://mlsanity.com/paper/2109.05488/) (2021)
- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement](https://mlsanity.com/paper/2205.07982/) (2022)
- [X-Avatar: Expressive Human Avatars](https://mlsanity.com/paper/2303.04805/) (2023)
- [HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2501.02973/) (2025)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration](https://mlsanity.com/paper/2103.02845/) (2021)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Towards unconstrained joint hand-object reconstruction from RGB videos](https://mlsanity.com/paper/2108.07044/) (2021)
- [Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution](https://mlsanity.com/paper/2204.13062/) (2022)
- [I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling](https://mlsanity.com/paper/2102.03725/) (2021)
- [Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans](https://mlsanity.com/paper/2312.00775/) (2023)
- [Zero-Shot Robot Manipulation from Passive Human Videos](https://mlsanity.com/paper/2302.02011/) (2023)

## Abstract (arXiv)

We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild.