# Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation

arXiv: https://arxiv.org/abs/1702.03431 · HTML: https://mlsanity.com/paper/1702.03431/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2017.

## Built on by (most cited first, 13 of 13)

- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [DART: Articulated Hand Model with Diverse Accessories and Rich Textures](https://mlsanity.com/paper/2210.07650/) (2022)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [Reconstructing Interacting Hands with Interaction Prior from Monocular Images](https://mlsanity.com/paper/2308.14082/) (2023)
- [HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models](https://mlsanity.com/paper/2406.01334/) (2024)
- [A hybrid classification-regression approach for 3D hand pose estimation using graph convolutional networks](https://mlsanity.com/paper/2105.10902/) (2021)
- [UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling](https://mlsanity.com/paper/2602.21631/) (2026)
- [Skeleton-aware multi-scale heatmap regression for 2D hand pose estimation](https://mlsanity.com/paper/2105.10904/) (2021)
- [Neural Variational Learning for Grounded Language Acquisition](https://mlsanity.com/paper/2107.14593/) (2021)
- [Robustness Analysis of Deep Learning Models for Population Synthesis](https://mlsanity.com/paper/2211.13339/) (2022)
- [Pushing the Envelope for Depth-Based Semi-Supervised 3D Hand Pose Estimation with Consistency Training](https://mlsanity.com/paper/2303.15147/) (2023)
- [EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View](https://mlsanity.com/paper/2509.13883/) (2025)
- [Deep Measurement Updates for Bayes Filters](https://mlsanity.com/paper/2112.00380/) (2021)

## Abstract (arXiv)

State-of-the-art methods for 3D hand pose estimation from depth images require large amounts of annotated training data. We propose to model the statistical relationships of 3D hand poses and corresponding depth images using two deep generative models with a shared latent space. By design, our architecture allows for learning from unlabeled image data in a semi-supervised manner. Assuming a one-to-one mapping between a pose and a depth map, any given point in the shared latent space can be projected into both a hand pose and a corresponding depth map. Regressing the hand pose can then be done by learning a discriminator to estimate the posterior of the latent pose given some depth maps. To improve generalization and to better exploit unlabeled depth maps, we jointly train a generator and a discriminator. At each iteration, the generator is updated with the back-propagated gradient from the discriminator to synthesize realistic depth maps of the articulated hand, while the discriminator benefits from an augmented training set of synthesized and unlabeled samples. The proposed discriminator network architecture is highly efficient and runs at 90 FPS on the CPU with accuracies comparable or better than state-of-art on 3 publicly available benchmarks.