mlsanityData to 7 Oct 2026

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

arXiv 2003.09572 ↗First posted 21 Mar 2020cs.CVcited by 106

Not on the map. Why?

Abstract (arXiv)

We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy. This is enabled by a new learning based architecture designed such that it can make use of all the sources of available hand training data: image data with either 2D or 3D annotations, as well as stand-alone 3D animations without corresponding image data. It features a 3D hand joint detection module and an inverse kinematics module which regresses not only 3D joint positions but also maps them to joint rotations in a single feed-forward pass. This output makes the method more directly usable for applications in computer vision and graphics compared to only regressing 3D joint positions. We demonstrate that our architectural design leads to a significant quantitative and qualitative improvement over the state of the art on several challenging benchmarks. Our model is publicly available for future research.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 25; 2023: 31; 2024: 20; 2025: 8; 2026: 5.

106 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  2. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  3. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  4. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  5. 2023Human Motion Generation: A Surveycited by 95
  6. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  7. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  8. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  9. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  10. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  11. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  12. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
38 more
  1. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  2. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  3. 2022AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstructioncited by 58
  4. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  5. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  6. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  7. 2021A Skeleton-Driven Neural Occupancy Representation for Articulated Handscited by 50
  8. 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 50
  9. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 48
  10. 2021Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstructioncited by 44
  11. 2023ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstructioncited by 37
  12. 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36
  13. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  14. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  15. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  16. 2021Hand Image Understanding via Deep Multi-Task Learningcited by 31
  17. 2022State of the Art in Dense Monocular Non-Rigid 3D Reconstructioncited by 30
  18. 2022Consistent 3D Hand Reconstruction in Video via self-supervised Learningcited by 28
  19. 2023Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapescited by 26
  20. 2022DART: Articulated Hand Model with Diverse Accessories and Rich Texturescited by 25
  21. 2021TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensorscited by 24
  22. 2023RelightableHands: Efficient Neural Relighting of Articulated Hand Modelscited by 21
  23. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  24. 2023SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Datacited by 20
  25. 2024Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimationcited by 19
  26. 2023Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglementcited by 18
  27. 2023Reconstructing Interacting Hands with Interaction Prior from Monocular Imagescited by 18
  28. 2022Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wildcited by 17
  29. 20233D Pose Estimation of Two Interacting Hands from a Monocular Event Cameracited by 17
  30. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  31. 2023DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Fieldcited by 16
  32. 2023SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstructioncited by 16
  33. 2021HandTailor: Towards High-Precision Monocular 3D Hand Recoverycited by 15
  34. 2025MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clipscited by 15
  35. 2024Complementing Event Streams and RGB Frames for Hand Mesh Reconstructioncited by 14
  36. 2024HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Modelscited by 14
  37. 2024HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactionscited by 13
  38. 2021Real-time RGBD-based Extended Body Pose Estimationcited by 12

50 most cited of 106 All 106 (JSON)