# 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal

arXiv: https://arxiv.org/abs/2207.11061 · HTML: https://mlsanity.com/paper/2207.11061/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [Capturing Hands in Action using Discriminative Salient Points and Physics Simulation](https://mlsanity.com/paper/1506.02178/) (2015)
- [Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering](https://mlsanity.com/paper/1904.04196/) (2019)
- [Hand Pose Estimation via Latent 2.5D Heatmap Regression](https://mlsanity.com/paper/1804.09534/) (2018)
- [Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data](https://mlsanity.com/paper/2003.09572/) (2020)
- [Semantic Amodal Segmentation](https://mlsanity.com/paper/1509.01329/) (2015)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [Cross-modal Deep Variational Hand Pose Estimation](https://mlsanity.com/paper/1803.11404/) (2018)
- [Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera](https://mlsanity.com/paper/2106.08059/) (2021)
- [Amodal Instance Segmentation](https://mlsanity.com/paper/1604.08202/) (2016)
- [SeGAN: Segmenting and Generating the Invisible](https://mlsanity.com/paper/1703.10239/) (2017)
- [Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation](https://mlsanity.com/paper/2107.00434/) (2021)
- [Human De-occlusion: Invisible Perception and Recovery for Humans](https://mlsanity.com/paper/2103.11597/) (2021)
- [Two-hand Global 3D Pose Estimation Using Monocular RGB](https://mlsanity.com/paper/2006.01320/) (2020)
- [Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements](https://mlsanity.com/paper/2111.00763/) (2021)
- [Visualizing the Invisible: Occluded Vehicle Segmentation and Recovery](https://mlsanity.com/paper/1907.09381/) (2019)
- [BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks](https://mlsanity.com/paper/2008.05079/) (2020)

## Built on by (most cited first, 30 of 36)

- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild](https://mlsanity.com/paper/2303.13652/) (2023)
- [Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos](https://mlsanity.com/paper/2209.09484/) (2022)
- [A Dataset of Relighted 3D Interacting Hands](https://mlsanity.com/paper/2310.17768/) (2023)
- [Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects](https://mlsanity.com/paper/2403.16428/) (2024)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [Reconstructing Interacting Hands with Interaction Prior from Monocular Images](https://mlsanity.com/paper/2308.14082/) (2023)
- [MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips](https://mlsanity.com/paper/2508.05506/) (2025)
- [RenderIH: A Large-scale Synthetic Dataset for 3D Interacting Hand Pose Estimation](https://mlsanity.com/paper/2309.09301/) (2023)
- [A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2304.03635/) (2023)
- [HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models](https://mlsanity.com/paper/2312.04867/) (2023)
- [HORT: Monocular Hand-held Objects Reconstruction with Transformers](https://mlsanity.com/paper/2503.21313/) (2025)
- [Recovering 3D Hand Mesh Sequence from a Single Blurry Image: A New Dataset and Temporal Unfolding](https://mlsanity.com/paper/2303.15417/) (2023)
- [AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild](https://mlsanity.com/paper/2407.18034/) (2024)
- [PEAR: Phrase-Based Hand-Object Interaction Anticipation](https://mlsanity.com/paper/2407.21510/) (2024)
- [You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-Person Multi-Task Human-Centric Perception](https://mlsanity.com/paper/2312.05525/) (2023)
- [Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2302.02410/) (2023)
- [Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidance](https://mlsanity.com/paper/2407.12307/) (2024)
- [3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands](https://mlsanity.com/paper/2401.00979/) (2024)
- [PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery](https://mlsanity.com/paper/2603.26068/) (2026)
- [Denoising Diffusion for 3D Hand Pose Estimation from Images](https://mlsanity.com/paper/2308.09523/) (2023)
- [V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction](https://mlsanity.com/paper/2409.13347/) (2024)
- [VM-BHINet:Vision Mamba Bimanual Hand Interaction Network for 3D Interacting Hand Mesh Recovery From a Single RGB Image](https://mlsanity.com/paper/2504.14618/) (2025)
- [Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands](https://mlsanity.com/paper/2608.11574/) (2026)
- [Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video](https://mlsanity.com/paper/2308.04074/) (2023)
- [Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks](https://mlsanity.com/paper/2404.05414/) (2024)
- [Monocular 3D Hand Mesh Recovery via Dual Noise Estimation](https://mlsanity.com/paper/2312.15916/) (2023)
- [Learning Transformation-Isomorphic Latent Space for Accurate Hand Pose Estimation](https://mlsanity.com/paper/2502.12535/) (2025)

## Abstract (arXiv)

Estimating 3D interacting hand pose from a single RGB image is essential for understanding human actions. Unlike most previous works that directly predict the 3D poses of two interacting hands simultaneously, we propose to decompose the challenging interacting hand pose estimation task and estimate the pose of each hand separately. In this way, it is straightforward to take advantage of the latest research progress on the single-hand pose estimation system. However, hand pose estimation in interacting scenarios is very challenging, due to (1) severe hand-hand occlusion and (2) ambiguity caused by the homogeneous appearance of hands. To tackle these two challenges, we propose a novel Hand De-occlusion and Removal (HDR) framework to perform hand de-occlusion and distractor removal. We also propose the first large-scale synthetic amodal hand dataset, termed Amodal InterHand Dataset (AIH), to facilitate model training and promote the development of the related research. Experiments show that the proposed method significantly outperforms previous state-of-the-art interacting hand pose estimation approaches. Codes and data are available at https://github.com/MengHao666/HDR.