mlsanityData to 14 Oct 2026

Towards unconstrained joint hand-object reconstruction from RGB videos

arXiv 2108.07044 ↗First posted 16 Aug 2021cs.CVcited by 58

Not on the map. Why?

Abstract (arXiv)

Our work aims to obtain 3D reconstruction of hands and manipulated objects from monocular videos. Reconstructing hand-object manipulations holds a great potential for robotics and learning from human demonstrations. The supervised learning approach to this problem, however, requires 3D supervision and remains limited to constrained laboratory settings and simulators for which 3D ground truth is available. In this paper we first propose a learning-free fitting approach for hand-object reconstruction which can seamlessly handle two-hand object interactions. Our method relies on cues obtained with common methods for object detection, hand pose estimation and instance segmentation. We quantitatively evaluate our approach and show that it can be applied to datasets with varying levels of difficulty for which training data is unavailable.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 56 of its 73 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

56 of 73 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  9. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  10. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
  11. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  12. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
38 more
  1. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  2. 2019PointRend: Image Segmentation as Renderingcited by 440
  3. 2017Neural 3D Mesh Renderercited by 437
  4. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  5. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  6. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  7. 2018Learning 3D Human Dynamics from Videocited by 316
  8. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  9. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
  10. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  11. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  12. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  13. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  14. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  15. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  16. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  17. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  18. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  19. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  20. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  21. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  22. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  23. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  24. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  25. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 147
  26. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  27. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  28. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  29. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  30. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  31. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  32. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  33. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  34. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  35. 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
  36. 2017Exploiting temporal information for 3D pose estimationcited by 89
  37. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  38. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 73

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 13; 2024: 13; 2025: 13; 2026: 7.

58 papers.

Sort:

By area:

  1. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
  2. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  3. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  4. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
  5. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  6. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  7. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70
  8. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  9. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 50
  10. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 37
  11. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  12. 2022Learning to Imitate Object Interactions from Internet Videoscited by 34
38 more
  1. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  2. 2022Interacting Hand-Object Pose Estimation via Dense Mutual Attentioncited by 22
  3. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 21
  4. 2022Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wildcited by 17
  5. 2025MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clipscited by 17
  6. 2022THOR-Net: End-to-end Graformer-based Realistic Two Hands and Object Reconstruction with Self-supervisioncited by 16
  7. 2023SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstructioncited by 16
  8. 2024Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamicscited by 15
  9. 2024HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fieldscited by 14
  10. 2024HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactionscited by 13
  11. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  12. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  13. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  14. 2024NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Modelcited by 10
  15. 2025BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splattingcited by 10
  16. 2022Stability-driven Contact Reconstruction From Monocular Color Imagescited by 8
  17. 2022S2Contact: Graph-based Network for 3D Hand-Object Contact Estimation with Semi-Supervised Learningcited by 8
  18. 2022UV-Based 3D Hand-Object Reconstruction with Grasp Optimizationcited by 8
  19. 2023DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulationcited by 8
  20. 2024Physics-aware Hand-object Interaction Denoisingcited by 8
  21. 2025Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidancecited by 8
  22. 2023DiffHand: End-to-End Hand Mesh Reconstruction via Diffusion Modelscited by 7
  23. 2023Neural Voting Field for Camera-Space 3D Hand Pose Estimationcited by 6
  24. 2024Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modelingcited by 6
  25. 2023PHRIT: Parametric Hand Representation with Implicit Templatecited by 5
  26. 2023In-Hand 3D Object Reconstruction from a Monocular RGB Videocited by 4
  27. 2024Single-View Scene Point Cloud Human Grasp Generationcited by 4
  28. 2023Nonrigid Object Contact Estimation With Regional Unwrapping Transformercited by 3
  29. 2023HandyPriors: Physically Consistent Perception of Hand-Object Interactions with Differentiable Priorscited by 3
  30. 2025PhysPose: Refining 6D Object Poses with Physical Constraintscited by 3
  31. 2025Occlusion-Aware 3D Hand-Object Pose Estimation with Masked AutoEncoderscited by 3
  32. 2025HOSt3R: Keypoint-free Hand-Object 3D Reconstruction from RGB imagescited by 3
  33. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 3
  34. 2026GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splattingcited by 3
  35. 2026Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactionscited by 3
  36. 2025FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidancecited by 2
  37. 2026Towards in-the-wild Egocentric 3D Hand-Object Pose Estimationcited by 2
  38. 2024NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interactioncited by 1

50 most cited of 58 All 58 (JSON)