# HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

arXiv: https://arxiv.org/abs/2107.01205 · HTML: https://mlsanity.com/paper/2107.01205/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 67 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction](https://mlsanity.com/paper/2004.13449/) (2020)
- [End-to-end Hand Mesh Recovery from a Monocular RGB Image](https://mlsanity.com/paper/1902.09305/) (2019)
- [HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation](https://mlsanity.com/paper/2004.00060/) (2020)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data](https://mlsanity.com/paper/2003.09572/) (2020)
- [V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/1711.07399/) (2017)
- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Image](https://mlsanity.com/paper/1908.09999/) (2019)
- [EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream](https://mlsanity.com/paper/2012.06475/) (2020)
- [BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis](https://mlsanity.com/paper/1704.02612/) (2017)
- [SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation](https://mlsanity.com/paper/2007.05168/) (2020)
- [Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction](https://mlsanity.com/paper/2003.13764/) (2020)
- [HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/2004.01588/) (2020)
- [Differentiable Event Stream Simulator for Non-Rigid 3D Tracking](https://mlsanity.com/paper/2104.15139/) (2021)
- [Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals](https://mlsanity.com/paper/1712.03917/) (2017)
- [DEMEA: Deep Mesh Autoencoders for Non-Rigidly Deforming Objects](https://mlsanity.com/paper/1905.10290/) (2019)
- [DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation](https://mlsanity.com/paper/1708.08325/) (2017)
- [Dense 3D Regression for Hand Pose Estimation](https://mlsanity.com/paper/1711.08996/) (2017)
- [AWR: Adaptive Weighting Regression for 3D Hand Pose Estimation](https://mlsanity.com/paper/2007.09590/) (2020)
- [Depth-based hand pose estimation: methods, data, and challenges](https://mlsanity.com/paper/1504.06378/) (2015)
- [IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/1904.12144/) (2019)

## Built on by (most cited first, 11 of 11)

- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [Scene-aware Egocentric 3D Human Pose Estimation](https://mlsanity.com/paper/2212.11684/) (2022)
- [High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition](https://mlsanity.com/paper/2307.05541/) (2023)
- [THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervision](https://mlsanity.com/paper/2210.13853/) (2022)
- [Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction](https://mlsanity.com/paper/2403.07346/) (2024)
- [ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction from a Single Depth Map](https://mlsanity.com/paper/2310.11811/) (2023)
- [HandDAGT: A Denoising Adaptive Graph Transformer for 3D Hand Pose Estimation](https://mlsanity.com/paper/2407.20542/) (2024)
- [Geometry-Aware 3D Salient Object Detection Network](https://mlsanity.com/paper/2502.16488/) (2025)
- [Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priors](https://mlsanity.com/paper/2508.09629/) (2025)
- [HandMCM: Multi-modal Point Cloud-based Correspondence State Space Model for 3D Hand Pose Estimation](https://mlsanity.com/paper/2602.01586/) (2026)
- [HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion](https://mlsanity.com/paper/2608.20093/) (2026)

## Abstract (arXiv)

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. Existing methods addressing it directly regress hand meshes via 2D convolutional neural networks, which leads to artefacts due to perspective distortions in the images. To address the limitations of the existing methods, we develop HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. The input to our network is a 3D voxelized-depth-map-based on the truncated signed distance function (TSDF). HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data. In extensive evaluations on three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and HO-3D, the proposed HandVoxNet++ achieves state-of-the-art performance. In this journal extension of our previous approach presented at CVPR 2020, we gain 41.09% and 13.7% higher shape alignment accuracy on SynHand5M and HANDS19 datasets, respectively. Our method is ranked first on the HANDS19 challenge dataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the submission of our results to the portal in August 2020.