# Disentangling Latent Hands for Image Synthesis and Pose Estimation

arXiv: https://arxiv.org/abs/1812.01002 · HTML: https://mlsanity.com/paper/1812.01002/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2018.

## Built on by (most cited first, 30 of 44)

- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis](https://mlsanity.com/paper/2109.05488/) (2021)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [A Skeleton-Driven Neural Occupancy Representation for Articulated Hands](https://mlsanity.com/paper/2109.11399/) (2021)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)
- [Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation](https://mlsanity.com/paper/2107.00434/) (2021)
- [Learning to Imitate Object Interactions from Internet Videos](https://mlsanity.com/paper/2211.13225/) (2022)
- [Hand Image Understanding via Deep Multi-Task Learning](https://mlsanity.com/paper/2107.11646/) (2021)
- [Directional Connectivity-based Segmentation of Medical Images](https://mlsanity.com/paper/2304.00145/) (2023)
- [Consistent 3D Hand Reconstruction in Video via self-supervised Learning](https://mlsanity.com/paper/2201.09548/) (2022)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements](https://mlsanity.com/paper/2111.00763/) (2021)
- [HMP: Hand Motion Priors for Pose and Shape Estimation from Video](https://mlsanity.com/paper/2312.16737/) (2023)
- [Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation](https://mlsanity.com/paper/2303.04991/) (2023)
- [Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning](https://mlsanity.com/paper/2503.00986/) (2025)
- [Reconstructing Interacting Hands with Interaction Prior from Monocular Images](https://mlsanity.com/paper/2308.14082/) (2023)
- [HandTailor: Towards High-Precision Monocular 3D Hand Recovery](https://mlsanity.com/paper/2102.09244/) (2021)
- [A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2304.03635/) (2023)
- [HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models](https://mlsanity.com/paper/2406.01334/) (2024)
- [UV-Based 3D Hand-Object Reconstruction with Grasp Optimization](https://mlsanity.com/paper/2211.13429/) (2022)
- [Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation](https://mlsanity.com/paper/2106.05954/) (2021)
- [HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning](https://mlsanity.com/paper/2302.00988/) (2023)
- [Local and Global Point Cloud Reconstruction for 3D Hand Pose Estimation](https://mlsanity.com/paper/2112.06389/) (2021)
- [End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2204.08154/) (2022)
- [Im2Mesh GAN: Accurate 3D Hand Mesh Recovery from a Single RGB Image](https://mlsanity.com/paper/2101.11239/) (2021)
- [RealisticHands: A Hybrid Model for 3D Hand Reconstruction](https://mlsanity.com/paper/2108.13995/) (2021)
- [3D Hand Pose and Shape Estimation from RGB Images for Keypoint-Based Hand Gesture Recognition](https://mlsanity.com/paper/2109.13879/) (2021)
- [HMD-EgoPose: Head-Mounted Display-Based Egocentric Marker-Less Tool and Hand Pose Estimation for Augmented Surgical Guidance](https://mlsanity.com/paper/2202.11891/) (2022)
- [Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidance](https://mlsanity.com/paper/2407.12307/) (2024)
- [Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environments](https://mlsanity.com/paper/2211.03587/) (2022)

## Abstract (arXiv)

Hand image synthesis and pose estimation from RGB images are both highly challenging tasks due to the large discrepancy between factors of variation ranging from image background content to camera viewpoint. To better analyze these factors of variation, we propose the use of disentangled representations and a disentangled variational autoencoder (dVAE) that allows for specific sampling and inference of these factors. The derived objective from the variational lower bound as well as the proposed training strategy are highly flexible, allowing us to handle cross-modal encoders and decoders as well as semi-supervised learning scenarios. Experiments show that our dVAE can synthesize highly realistic images of the hand specifiable by both pose and image background content and also estimate 3D hand poses from RGB images with accuracy competitive with state-of-the-art on two public benchmarks.