mlsanityData to 7 Oct 2026

Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

arXiv 1506.02178 ↗First posted 6 Jun 2015cs.CVcited by 115

Not on the map. Why?

Abstract (arXiv)

Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 23; 2023: 31; 2024: 22; 2025: 14; 2026: 6.

115 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  2. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  3. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  4. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  5. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  6. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  7. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  8. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  9. 2021LEAP: Learning Articulated Occupancy of Peoplecited by 107
  10. 20233D Human Pose Estimation via Intuitive Physicscited by 100
  11. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  12. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
38 more
  1. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  2. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  3. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  4. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  5. 2021SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognitioncited by 59
  6. 2022COAP: Compositional Articulated Occupancy of Peoplecited by 59
  7. 2023SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understandingcited by 59
  8. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  9. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  10. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 52
  11. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  12. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 46
  13. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  14. 2021Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentationcited by 40
  15. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  16. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  17. 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36
  18. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  19. 2023A Dataset of Relighted 3D Interacting Handscited by 35
  20. 2024RealDex: Towards Human-like Grasping for Robotic Dexterous Handcited by 33
  21. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  22. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  23. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  24. 2021Contact-Aware Retargeting of Skinned Motioncited by 31
  25. 2024InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusioncited by 28
  26. 2021Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinementscited by 27
  27. 2023GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistencycited by 26
  28. 2021RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Videocited by 23
  29. 2022PressureVision: Estimating Hand Pressure from a Single RGB Imagecited by 23
  30. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  31. 2023RelightableHands: Efficient Neural Relighting of Articulated Hand Modelscited by 21
  32. 2022Physical Interaction: Reconstructing Hand-object Interactions with Physicscited by 19
  33. 2023Reconstructing Interacting Hands with Interaction Prior from Monocular Imagescited by 18
  34. 2024Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaptioncited by 18
  35. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 18
  36. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  37. 2023EvHandPose: Event-based 3D Hand Pose Estimation with Sparse Supervisioncited by 17
  38. 20233D Pose Estimation of Two Interacting Hands from a Monocular Event Cameracited by 17

50 most cited of 115 All 115 (JSON)