# DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling

arXiv: https://arxiv.org/abs/2008.08213 · HTML: https://mlsanity.com/paper/2008.08213/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2020.

## Built on by (most cited first, 30 of 49)

- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [Expressive Whole-Body 3D Gaussian Avatar](https://mlsanity.com/paper/2407.21686/) (2024)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)
- [LISA: Learning Implicit Shape and Appearance of Hands](https://mlsanity.com/paper/2204.01695/) (2022)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution](https://mlsanity.com/paper/2204.13062/) (2022)
- [I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling](https://mlsanity.com/paper/2102.03725/) (2021)
- [A Skeleton-Driven Neural Occupancy Representation for Articulated Hands](https://mlsanity.com/paper/2109.11399/) (2021)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)
- [Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation](https://mlsanity.com/paper/2107.00434/) (2021)
- [Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video](https://mlsanity.com/paper/2211.12782/) (2022)
- [ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction](https://mlsanity.com/paper/2303.05938/) (2023)
- [Contrastive Representation Learning for Hand Shape Estimation](https://mlsanity.com/paper/2106.04324/) (2021)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [SUPR: A Sparse Unified Part-Based Human Representation](https://mlsanity.com/paper/2210.13861/) (2022)
- [RelightableHands: Efficient Neural Relighting of Articulated Hand Models](https://mlsanity.com/paper/2302.04866/) (2023)
- [High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition](https://mlsanity.com/paper/2307.05541/) (2023)
- [3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera](https://mlsanity.com/paper/2312.14157/) (2023)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging](https://mlsanity.com/paper/2106.10893/) (2021)
- [Authentic Hand Avatar from a Phone Scan via Universal Hand Model](https://mlsanity.com/paper/2405.07933/) (2024)
- [Identity-Aware Hand Mesh Estimation and Personalization from RGB Images](https://mlsanity.com/paper/2209.10840/) (2022)
- [Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images](https://mlsanity.com/paper/2308.11015/) (2023)
- [Spectrum AUC Difference (SAUCD): Human-aligned 3D Shape Evaluation](https://mlsanity.com/paper/2403.01619/) (2024)
- [Stability-driven Contact Reconstruction From Monocular Color Images](https://mlsanity.com/paper/2205.00848/) (2022)
- [Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2302.02410/) (2023)
- [Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation](https://mlsanity.com/paper/2106.05954/) (2021)

## Abstract (arXiv)

Human hands play a central role in interacting with other people and objects. For realistic replication of such hand motions, high-fidelity hand meshes have to be reconstructed. In this study, we firstly propose DeepHandMesh, a weakly-supervised deep encoder-decoder framework for high-fidelity hand mesh modeling. We design our system to be trained in an end-to-end and weakly-supervised manner; therefore, it does not require groundtruth meshes. Instead, it relies on weaker supervisions such as 3D joint coordinates and multi-view depth maps, which are easier to get than groundtruth meshes and do not dependent on the mesh topology. Although the proposed DeepHandMesh is trained in a weakly-supervised way, it provides significantly more realistic hand mesh than previous fully-supervised hand models. Our newly introduced penetration avoidance loss further improves results by replicating physical interaction between hand parts. Finally, we demonstrate that our system can also be applied successfully to the 3D hand mesh estimation from general images. Our hand model, dataset, and codes are publicly available at https://mks0601.github.io/DeepHandMesh/.