# Dual Grid Net: hand mesh vertex regression from single depth maps

arXiv: https://arxiv.org/abs/1907.10695 · HTML: https://mlsanity.com/paper/1907.10695/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2019.

## Built on by (most cited first, 10 of 10)

- [A Skeleton-Driven Neural Occupancy Representation for Articulated Hands](https://mlsanity.com/paper/2109.11399/) (2021)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)
- [Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes](https://mlsanity.com/paper/2302.14348/) (2023)
- [Reconstructing Interacting Hands with Interaction Prior from Monocular Images](https://mlsanity.com/paper/2308.14082/) (2023)
- [HandTailor: Towards High-Precision Monocular 3D Hand Recovery](https://mlsanity.com/paper/2102.09244/) (2021)
- [FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow](https://mlsanity.com/paper/2307.08100/) (2023)
- [UV-Based 3D Hand-Object Reconstruction with Grasp Optimization](https://mlsanity.com/paper/2211.13429/) (2022)
- [Local and Global Point Cloud Reconstruction for 3D Hand Pose Estimation](https://mlsanity.com/paper/2112.06389/) (2021)
- [Image-free Domain Generalization via CLIP for 3D Hand Pose Estimation](https://mlsanity.com/paper/2210.16788/) (2022)
- [SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation](https://mlsanity.com/paper/2402.01389/) (2024)

## Abstract (arXiv)

We present a method for recovering the dense 3D surface of the hand by regressing the vertex coordinates of a mesh model from a single depth map. To this end, we use a two-stage 2D fully convolutional network architecture. In the first stage, the network estimates a dense correspondence field for every pixel on the depth map or image grid to the mesh grid. In the second stage, we design a differentiable operator to map features learned from the previous stage and regress a 3D coordinate map on the mesh grid. Finally, we sample from the mesh grid to recover the mesh vertices, and fit it an articulated template mesh in closed form. During inference, the network can predict all the mesh vertices, transformation matrices for every joint and the joint coordinates in a single forward pass. When given supervision on the sparse key-point coordinates, our method achieves state-of-the-art accuracy on NYU dataset for key point localization while recovering mesh vertices and a dense correspondence map. Our framework can also be learned through self-supervision by minimizing a set of data fitting and kinematic prior terms. With multi-camera rig during training to resolve self-occlusion, it can perform competitively with strongly supervised methods Without any human annotation.