mlsanityData to 14 Oct 2026

Learning joint reconstruction of hands and manipulated objects

arXiv 1904.05767 ↗First posted 11 Apr 2019cs.CVcited by 378

Not on the map. Why?

Abstract (arXiv)

Estimating hand-object manipulations is essential for interpreting and imitating human actions. Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation. Yet, reconstructing hands and objects during manipulation is a more challenging task due to significant occlusions of both the hand and object. While presenting challenges, manipulations may also simplify the problem since the physics of contact restricts the space of valid hand-object configurations. For example, during manipulation, the hand and object should be in contact but not interpenetrate. In this work, we regularize the joint reconstruction of hands and objects with manipulation constraints. We present an end-to-end learnable model that exploits a novel contact loss that favors physically plausible hand-object constellations. Our approach improves grasp quality metrics over baselines, using RGB images as input. To train and evaluate the model, we also propose a new large-scale synthetic dataset, ObMan, with hand-object manipulations. We demonstrate the transferability of ObMan-trained models to real data.

Key facts. Cited by 378 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 48; 2022: 66; 2023: 86; 2024: 79; 2025: 52; 2026: 47.

378 papers.

Sort:

By area:

  1. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  2. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  3. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  4. 2023Reconstructing Hands in 3D with Transformerscited by 306
  5. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  6. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  7. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  8. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  9. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  10. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 153
  11. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
  12. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
38 more
  1. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  2. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  3. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  4. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  5. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 137
  6. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
  7. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 129
  8. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  9. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  10. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 124
  11. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  12. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 113
  13. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  14. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  15. 20233D Human Pose Estimation via Intuitive Physicscited by 103
  16. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  17. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 101
  18. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 96
  19. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  20. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
  21. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  22. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 89
  23. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 88
  24. 2023An Outlook into the Future of Egocentric Visioncited by 84
  25. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 82
  26. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  27. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  28. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
  29. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  30. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  31. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  32. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70
  33. 2023X-Avatar: Expressive Human Avatarscited by 66
  34. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 66
  35. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  36. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 64
  37. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  38. 2022LISA: Learning Implicit Shape and Appearance of Handscited by 63

50 most cited of 378 All 378 (JSON)