mlsanityData to 7 Oct 2026

Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation

arXiv 2107.00434 ↗First posted 1 Jul 2021cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

In natural conversation and interaction, our hands often overlap or are in contact with each other. Due to the homogeneous appearance of hands, this makes estimating the 3D pose of interacting hands from images difficult. In this paper we demonstrate that self-similarity, and the resulting ambiguities in assigning pixel observations to the respective hands and their parts, is a major cause of the final 3D pose error. Motivated by this insight, we propose DIGIT, a novel method for estimating the 3D poses of two interacting hands from a single monocular image. The method consists of two interwoven branches that process the input imagery into a per-pixel semantic part segmentation mask and a visual feature volume. In contrast to prior work, we do not decouple the segmentation from the pose estimation stage, but rather leverage the per-pixel probabilities directly in the downstream pose estimation task. To do so, the part probabilities are merged with the visual features and processed via fully-convolutional layers. We experimentally show that the proposed approach achieves new state-of-the-art performance on the InterHand2.6M dataset. We provide detailed ablation studies to demonstrate the efficacy of our method and to provide insights into how the modelling of pixel ownership affects 3D hand pose estimation.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 44 of its 66 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

44 of 66 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2017Neural 3D Mesh Renderercited by 437
  10. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  11. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  12. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
32 more
  1. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  2. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  3. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  4. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  5. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  6. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  7. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  8. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  9. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  11. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  12. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  13. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  14. 2020Epipolar Transformerscited by 117
  15. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  16. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  17. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  18. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  19. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  20. 2020Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flowscited by 94
  21. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  22. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  23. 2017Human Pose Regression by Combining Indirect Part Detection and Contextual Informationcited by 52
  24. 2020DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modelingcited by 49
  25. 2018Disentangling Latent Hands for Image Synthesis and Pose Estimationcited by 44
  26. 2017Using a single RGB frame for real time 3D hand pose estimation in the wildcited by 39
  27. 2020Two-hand Global 3D Pose Estimation Using Monocular RGBcited by 27
  28. 2015Hands Deep in Deep Learning for Hand Pose Estimationcited by 20
  29. 2017DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimationcited by 19
  30. 2017A Dual-Source Approach for 3D Human Pose Estimation from a Single Imagecited by 7
  31. 2015Hand Pose Estimation through Semi-Supervised and Weakly-Supervised Learningcited by 6
  32. 2016Hand Segmentation for Hand-Object Interaction from Depth mapcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 22; 2024: 5; 2025: 5; 2026: 2.

40 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  2. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  3. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  4. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  5. 2023ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstructioncited by 37
  6. 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36
  7. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  8. 2023A Dataset of Relighted 3D Interacting Handscited by 35
  9. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  10. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  11. 2023Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapescited by 26
  12. 2023HMP: Hand Motion Priors for Pose and Shape Estimation from Videocited by 23
28 more
  1. 2023Reconstructing Interacting Hands with Interaction Prior from Monocular Imagescited by 18
  2. 2022TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learningcited by 17
  3. 20233D Pose Estimation of Two Interacting Hands from a Monocular Event Cameracited by 17
  4. 2023RenderIH: A Large-scale Synthetic Dataset for 3D Interacting Hand Pose Estimationcited by 16
  5. 2023A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 15
  6. 2023HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Modelscited by 13
  7. 2025Learning Dense Hand Contact Estimation from Imbalanced Datacited by 12
  8. 2023FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flowcited by 10
  9. 2023Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Imagescited by 9
  10. 2023Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Imagecited by 8
  11. 2023MeMaHand: Exploiting Mesh-Mano Interaction for Single Image Two-Hand Reconstructioncited by 8
  12. 2023HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learningcited by 7
  13. 2023Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recoverycited by 7
  14. 2022LWA-HAND: Lightweight Attention Hand for Interacting Hand Reconstructioncited by 5
  15. 2022HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flowcited by 5
  16. 2024Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidancecited by 5
  17. 2024Fine-Grained Multi-View Hand Reconstruction Using Inverse Renderingcited by 3
  18. 2025PALM: A Dataset and Baseline for Learning Multi-subject Hand Priorcited by 3
  19. 2023Denoising Diffusion for 3D Hand Pose Estimation from Imagescited by 2
  20. 2025VM-BHINet:Vision Mamba Bimanual Hand Interaction Network for 3D Interacting Hand Mesh Recovery From a Single RGB Imagecited by 2
  21. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learningcited by 2
  22. 2023UltraGlove: Hand Pose Estimation with Mems-Ultrasonic Sensorscited by 1
  23. 2023Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Videocited by 1
  24. 2024Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networkscited by 1
  25. 2025Visuo-Acoustic Hand Pose and Contact Estimationcited by 1
  26. 2026Training-Free Dense Hand Contact Estimation with Multi-Modal Large Language Modelscited by 1
  27. 2023HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimationcited by 0
  28. 2025DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactionscited by 0