# Augmented Skeleton Space Transfer for Depth-based Hand Pose Estimation

arXiv: https://arxiv.org/abs/1805.04497 · HTML: https://mlsanity.com/paper/1805.04497/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2018.

## Built on by (most cited first, 15 of 15)

- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [TriHorn-Net: A Model for Accurate Depth-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2206.07117/) (2022)
- [A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets](https://mlsanity.com/paper/2204.03328/) (2022)
- [MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild](https://mlsanity.com/paper/2412.13393/) (2024)
- [Image-free Domain Generalization via CLIP for 3D Hand Pose Estimation](https://mlsanity.com/paper/2210.16788/) (2022)
- [QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects](https://mlsanity.com/paper/2502.19769/) (2025)
- [Transformer-based Action recognition in hand-object interacting scenarios](https://mlsanity.com/paper/2210.11387/) (2022)
- [Transformer-based Global 3D Hand Pose Estimation in Two Hands Manipulating Objects Scenarios](https://mlsanity.com/paper/2210.11384/) (2022)
- [Pushing the Envelope for Depth-Based Semi-Supervised 3D Hand Pose Estimation with Consistency Training](https://mlsanity.com/paper/2303.15147/) (2023)
- [Geometric Synthesis: A Free lunch for Large-scale Palmprint Recognition Model Pretraining](https://mlsanity.com/paper/2203.05703/) (2022)
- [HOReeNet: 3D-aware Hand-Object Grasping Reenactment](https://mlsanity.com/paper/2211.06195/) (2022)
- [Leveraging 2D Masked Reconstruction for Domain Adaptation of 3D Pose Estimation](https://mlsanity.com/paper/2501.08408/) (2025)

## Abstract (arXiv)

Crucial to the success of training a depth-based 3D hand pose estimator (HPE) is the availability of comprehensive datasets covering diverse camera perspectives, shapes, and pose variations. However, collecting such annotated datasets is challenging. We propose to complete existing databases by generating new database entries. The key idea is to synthesize data in the skeleton space (instead of doing so in the depth-map space) which enables an easy and intuitive way of manipulating data entries. Since the skeleton entries generated in this way do not have the corresponding depth map entries, we exploit them by training a separate hand pose generator (HPG) which synthesizes the depth map from the skeleton entries. By training the HPG and HPE in a single unified optimization framework enforcing that 1) the HPE agrees with the paired depth and skeleton entries; and 2) the HPG-HPE combination satisfies the cyclic consistency (both the input and the output of HPG-HPE are skeletons) observed via the newly generated unpaired skeletons, our algorithm constructs a HPE which is robust to variations that go beyond the coverage of the existing database. Our training algorithm adopts the generative adversarial networks (GAN) training process. As a by-product, we obtain a hand pose discriminator (HPD) that is capable of picking out realistic hand poses. Our algorithm exploits this capability to refine the initial skeleton estimates in testing, further improving the accuracy. We test our algorithm on four challenging benchmark datasets (ICVL, MSRA, NYU and Big Hand 2.2M datasets) and demonstrate that our approach outperforms or is on par with state-of-the-art methods quantitatively and qualitatively.