mlsanityData to 14 Oct 2026

Reconstructing Hand-Object Interactions in the Wild

arXiv 2012.09856 ↗First posted 17 Dec 2020cs.CVcited by 144

Not on the map. Why?

Abstract (arXiv)

In this work we explore reconstructing hand-object interactions in the wild. The core challenge of this problem is the lack of appropriate 3D labeled data. To overcome this issue, we propose an optimization-based procedure which does not require direct 3D supervision. The general strategy we adopt is to exploit all available related data (2D bounding boxes, 2D hand keypoints, 2D instance masks, 3D object models, 3D in-the-lab MoCap) to provide constraints for the 3D reconstruction. Rather than optimizing the hand and object individually, we optimize them jointly which allows us to impose additional constraints based on hand-object contact, collision, and occlusion. Our method produces compelling reconstructions on the challenging in-the-wild data from the EPIC Kitchens and the 100 Days of Hands datasets, across a range of object categories. Quantitatively, we demonstrate that our approach compares favorably to existing approaches in the lab settings where ground truth 3D annotations are available.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 25; 2023: 34; 2024: 28; 2025: 24; 2026: 23.

144 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  2. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  3. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 148
  4. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
  5. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  6. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 113
  7. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  8. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 89
  9. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 82
  10. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  11. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  12. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
38 more
  1. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  2. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  3. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  4. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70
  5. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 66
  6. 2022LISA: Learning Implicit Shape and Appearance of Handscited by 63
  7. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 60
  8. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 58
  9. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 57
  10. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  11. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 56
  12. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  13. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  14. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  15. 2021Human Hands as Probes for Interactive Object Understandingcited by 52
  16. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 50
  17. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 49
  18. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41
  19. 2023Detecting Human-Object Contact in Imagescited by 41
  20. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  21. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 37
  22. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Datacited by 37
  23. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 36
  24. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  25. 2022Learning to Imitate Object Interactions from Internet Videoscited by 34
  26. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 34
  27. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 33
  28. 2022Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Graspscited by 29
  29. 2022Consistent 3D Hand Reconstruction in Video via self-supervised Learningcited by 28
  30. 2022HARP: Personalized Hand Reconstruction from a Monocular RGB Videocited by 28
  31. 2023Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapescited by 26
  32. 2022Visual Object Tracking in First Person Visioncited by 25
  33. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  34. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 23
  35. 2021Is First Person Vision Challenging for Object Tracking?cited by 21
  36. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  37. 2022Hand-Object Interaction Image Generationcited by 19
  38. 2023Decaf: Monocular Deformation Capture for Face and Hand Interactionscited by 18

50 most cited of 144 All 144 (JSON)