# HandTailor: Towards High-Precision Monocular 3D Hand Recovery

arXiv: https://arxiv.org/abs/2102.09244 · HTML: https://mlsanity.com/paper/2102.09244/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)
- [Hand Pose Estimation via Latent 2.5D Heatmap Regression](https://mlsanity.com/paper/1804.09534/) (2018)
- [Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data](https://mlsanity.com/paper/2003.09572/) (2020)
- [Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints](https://mlsanity.com/paper/2003.09282/) (2020)
- [Cross-modal Deep Variational Hand Pose Estimation](https://mlsanity.com/paper/1803.11404/) (2018)
- [Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor](https://mlsanity.com/paper/1704.02201/) (2017)
- [Disentangling Latent Hands for Image Synthesis and Pose Estimation](https://mlsanity.com/paper/1812.01002/) (2018)
- [Using a single RGB frame for real time 3D hand pose estimation in the wild](https://mlsanity.com/paper/1712.03866/) (2017)
- [Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge](https://mlsanity.com/paper/2004.00176/) (2020)
- [Single Image 3D Hand Reconstruction with Mesh Convolutions](https://mlsanity.com/paper/1905.01326/) (2019)
- [Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs](https://mlsanity.com/paper/1606.07253/) (2016)
- [HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/2004.01588/) (2020)
- [BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks](https://mlsanity.com/paper/2008.05079/) (2020)
- [Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals](https://mlsanity.com/paper/1712.03917/) (2017)
- [Depth-based hand pose estimation: methods, data, and challenges](https://mlsanity.com/paper/1504.06378/) (2015)
- [Dual Grid Net: hand mesh vertex regression from single depth maps](https://mlsanity.com/paper/1907.10695/) (2019)

## Built on by (most cited first, 15 of 15)

- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion](https://mlsanity.com/paper/2403.19417/) (2024)
- [ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis](https://mlsanity.com/paper/2109.05488/) (2021)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction](https://mlsanity.com/paper/2303.05938/) (2023)
- [HORT: Monocular Hand-held Objects Reconstruction with Transformers](https://mlsanity.com/paper/2503.21313/) (2025)
- [Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition](https://mlsanity.com/paper/2407.00299/) (2024)
- [MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints](https://mlsanity.com/paper/2404.10227/) (2024)
- [DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation](https://mlsanity.com/paper/2310.07206/) (2023)
- [End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2204.08154/) (2022)
- [In-Hand 3D Object Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2312.16425/) (2023)
- [Semi-supervised Hand Appearance Recovery via Structure Disentanglement and Dual Adversarial Discrimination](https://mlsanity.com/paper/2303.06380/) (2023)
- [Lightweight Estimation of Hand Mesh and Biomechanically Feasible Kinematic Parameters](https://mlsanity.com/paper/2303.14838/) (2023)
- [HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB images](https://mlsanity.com/paper/2405.01066/) (2024)
- [EHPE: A Segmented Architecture for Enhanced Hand Pose Estimation](https://mlsanity.com/paper/2507.09560/) (2025)

## Abstract (arXiv)

3D hand pose estimation and shape recovery are challenging tasks in computer vision. We introduce a novel framework HandTailor, which combines a learning-based hand module and an optimization-based tailor module to achieve high-precision hand mesh recovery from a monocular RGB image. The proposed hand module unifies perspective projection and weak perspective projection in a single network towards accuracy-oriented and in-the-wild scenarios. The proposed tailor module then utilizes the coarsely reconstructed mesh model provided by the hand module as initialization, and iteratively optimizes an energy function to obtain better results. The tailor module is time-efficient, costs only 8ms per frame on a modern CPU. We demonstrate that HandTailor can get state-of-the-art performance on several public benchmarks, with impressive qualitative results on in-the-wild experiments. Code and video are available on our project webpage https://sites.google.com/view/handtailor.