mlsanityData to 7 Oct 2026

Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements

arXiv 2111.00763 ↗First posted 1 Nov 2021cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

3D interacting hand reconstruction is essential to facilitate human-machine interaction and human behaviors understanding. Previous works in this field either rely on auxiliary inputs such as depth images or they can only handle a single hand if monocular single RGB images are used. Single-hand methods tend to generate collided hand meshes, when applied to closely interacting hands, since they cannot model the interactions between two hands explicitly. In this paper, we make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions. This is made possible via a two-stage framework. Specifically, the first stage adopts a convolutional neural network to generate coarse predictions that tolerate collisions but encourage pose-accurate hand meshes. The second stage progressively ameliorates the collisions through a series of factorized refinements while retaining the preciseness of 3D poses. We carefully investigate potential implementations for the factorized refinement, considering the trade-off between efficiency and accuracy. Extensive quantitative and qualitative results on large-scale datasets such as InterHand2.6M demonstrate the effectiveness of the proposed approach.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

37 of 52 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  5. 2017End-to-end Recovery of Human Shape and Posecited by 883
  6. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  7. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  8. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  9. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  10. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  11. 2017Integral Human Pose Regressioncited by 255
  12. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
25 more
  1. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  2. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  3. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  4. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  5. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  6. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  7. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  8. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  9. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  10. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  11. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  12. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  13. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  14. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  15. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  16. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  17. 2019Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interactioncited by 48
  18. 2018Disentangling Latent Hands for Image Synthesis and Pose Estimationcited by 44
  19. 2019Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wildcited by 38
  20. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34
  21. 2016Fast and Robust Hand Tracking Using Detection-Guided Optimizationcited by 30
  22. 2016Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNscited by 29
  23. 2020BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networkscited by 23
  24. 2020Chasing the Tail in Monocular 3D Human Reconstruction with Prototype Memorycited by 11
  25. 2016Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimationcited by 8

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 13; 2024: 5; 2025: 3.

27 papers.

Sort:

By area:

  1. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  2. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  3. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  4. 2023HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recoverycited by 48
  5. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  6. 2023ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstructioncited by 37
  7. 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36
  8. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  9. 2023A Dataset of Relighted 3D Interacting Handscited by 35
  10. 2024InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusioncited by 28
  11. 2023Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapescited by 26
  12. 2021Not All Voxels Are Equal: Semantic Scene Completion from the Point-Voxel Perspectivecited by 22
15 more
  1. 2023Reconstructing Interacting Hands with Interaction Prior from Monocular Imagescited by 18
  2. 2024Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaptioncited by 18
  3. 2023A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 15
  4. 2023HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Modelscited by 13
  5. 2024AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wildcited by 10
  6. 2023Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Imagescited by 9
  7. 2024DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Imagecited by 9
  8. 2023Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Imagecited by 8
  9. 2022LWA-HAND: Lightweight Attention Hand for Interacting Hand Reconstructioncited by 5
  10. 2023Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstructioncited by 4
  11. 2025Reconstructing Close Human Interaction with Appearance and Proxemics Reasoningcited by 4
  12. 2025PhysPose: Refining 6D Object Poses with Physical Constraintscited by 3
  13. 2025VM-BHINet:Vision Mamba Bimanual Hand Interaction Network for 3D Interacting Hand Mesh Recovery From a Single RGB Imagecited by 2
  14. 2023UltraGlove: Hand Pose Estimation with Mems-Ultrasonic Sensorscited by 1
  15. 2024Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networkscited by 1