# Hand Pose Estimation through Semi-Supervised and Weakly-Supervised Learning

arXiv: https://arxiv.org/abs/1511.06728 · HTML: https://mlsanity.com/paper/1511.06728/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2015.

## Built on by (most cited first, 6 of 6)

- [Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation](https://mlsanity.com/paper/2107.00434/) (2021)
- [Domain Adaptive Hand Keypoint and Pixel Localization in the Wild](https://mlsanity.com/paper/2203.08344/) (2022)
- [3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions](https://mlsanity.com/paper/2404.04823/) (2024)
- [AugmentGest: Can Random Data Cropping Augmentation Boost Gesture Recognition Performance?](https://mlsanity.com/paper/2506.07216/) (2025)
- [End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2204.08154/) (2022)
- [Universal Domain Adaptation in Ordinal Regression](https://mlsanity.com/paper/2106.11576/) (2021)

## Abstract (arXiv)

We propose a method for hand pose estimation based on a deep regressor trained on two different kinds of input. Raw depth data is fused with an intermediate representation in the form of a segmentation of the hand into parts. This intermediate representation contains important topological information and provides useful cues for reasoning about joint locations. The mapping from raw depth to segmentation maps is learned in a semi/weakly-supervised way from two different datasets: (i) a synthetic dataset created through a rendering pipeline including densely labeled ground truth (pixelwise segmentations); and (ii) a dataset with real images for which ground truth joint positions are available, but not dense segmentations. Loss for training on real images is generated from a patch-wise restoration process, which aligns tentative segmentation maps with a large dictionary of synthetic poses. The underlying premise is that the domain shift between synthetic and real data is smaller in the intermediate representation, where labels carry geometric and topological meaning, than in the raw input domain. Experiments on the NYU dataset show that the proposed training method decreases error on joints over direct regression of joints from depth data by 15.7%.