# Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild

arXiv: https://arxiv.org/abs/2004.01946 · HTML: https://mlsanity.com/paper/2004.01946/

## Key facts

- Cited by 146 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2020.

## Built on by (most cited first, 30 of 146)

- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization](https://mlsanity.com/paper/2303.14189/) (2023)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation](https://mlsanity.com/paper/2104.14639/) (2021)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [Interacting Attention Graph for Single Image Two-Hand Reconstruction](https://mlsanity.com/paper/2203.09364/) (2022)
- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [LISA: Learning Implicit Shape and Appearance of Hands](https://mlsanity.com/paper/2204.01695/) (2022)
- [Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration](https://mlsanity.com/paper/2103.02845/) (2021)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [Towards unconstrained joint hand-object reconstruction from RGB videos](https://mlsanity.com/paper/2108.07044/) (2021)
- [Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution](https://mlsanity.com/paper/2204.13062/) (2022)
- [I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling](https://mlsanity.com/paper/2102.03725/) (2021)
- [Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans](https://mlsanity.com/paper/2312.00775/) (2023)
- [Zero-Shot Robot Manipulation from Passive Human Videos](https://mlsanity.com/paper/2302.02011/) (2023)
- [A Skeleton-Driven Neural Occupancy Representation for Articulated Hands](https://mlsanity.com/paper/2109.11399/) (2021)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery](https://mlsanity.com/paper/2304.05690/) (2023)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)

## Abstract (arXiv)

We introduce a simple and effective network architecture for monocular 3D hand pose estimation consisting of an image encoder followed by a mesh convolutional decoder that is trained through a direct 3D hand mesh reconstruction loss. We train our network by gathering a large-scale dataset of hand action in YouTube videos and use it as a source of weak supervision. Our weakly-supervised mesh convolutions-based system largely outperforms state-of-the-art methods, even halving the errors on the in the wild benchmark. The dataset and additional resources are available at https://arielai.com/mesh_hands.