mlsanityData to 7 Oct 2026

What's in your hands? 3D Reconstruction of Generic Objects in Hands

arXiv 2204.07153 ↗First posted 14 Apr 2022cs.CVcited by 90

Not on the map. Why?

Abstract (arXiv)

Our work aims to reconstruct hand-held objects given a single RGB image. In contrast to prior works that typically assume known 3D templates and reduce the problem to 3D pose estimation, our work reconstructs generic hand-held object without knowing their 3D templates. Our key insight is that hand articulation is highly predictive of the object shape, and we propose an approach that conditionally reconstructs the object based on the articulation and the visual input. Given an image depicting a hand-held object, we first use off-the-shelf systems to estimate the underlying hand pose and then infer the object shape in a normalized hand-centric coordinate frame. We parameterized the object by signed distance which are inferred by an implicit network which leverages the information from both visual feature and articulation-aware coordinates to process a query point. We perform experiments across three datasets and show that our method consistently outperforms baselines and is able to reconstruct a diverse set of objects. We analyze the benefits and robustness of explicit articulation conditioning and also show that this allows the hand pose estimation to further improve in test-time optimization.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 54 of its 75 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

54 of 75 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  7. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  8. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  9. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  12. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
38 more
  1. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  2. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  3. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  4. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  5. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  6. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  7. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  8. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  9. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  10. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  11. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  12. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  13. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  14. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  15. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  16. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  17. 2019CvxNet: Learnable Convex Decompositioncited by 173
  18. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  19. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  20. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  21. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  22. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  23. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  24. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  25. 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
  26. 2020Self-supervised Single-view 3D Reconstruction via Semantic Consistencycited by 127
  27. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  28. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  29. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  30. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  31. 2020Shape and Viewpoint without Keypointscited by 92
  32. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  33. 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
  34. 2021Shelf-Supervised Mesh Prediction in the Wildcited by 65
  35. 2014Category-Specific Object Reconstruction from a Single Imagecited by 56
  36. 2016Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Inputcited by 56
  37. 2017Using a single RGB frame for real time 3D hand pose estimation in the wildcited by 39
  38. 2020Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusioncited by 39

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 26; 2024: 19; 2025: 23; 2026: 15.

90 papers.

Sort:

By area:

  1. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  2. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  3. 2022Full-Body Articulated Human-Object Interactioncited by 76
  4. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  5. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  6. 2022AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstructioncited by 58
  7. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  8. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  9. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 49
  10. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  11. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  12. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 36
38 more
  1. 2022Reconstructing Hand-Held Objects from Monocular Videocited by 35
  2. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  3. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  4. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  5. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  6. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  7. 2022State of the Art in Dense Monocular Non-Rigid 3D Reconstructioncited by 30
  8. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  9. 2022In-Hand 3D Object Scanning from an RGB Sequencecited by 27
  10. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  11. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 22
  12. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  13. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  14. 2022Hand-Object Interaction Image Generationcited by 19
  15. 2024Controlling the World by Sleight of Handcited by 19
  16. 2023DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Fieldcited by 16
  17. 2023Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Viewscited by 16
  18. 2023SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstructioncited by 16
  19. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  20. 2025MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clipscited by 15
  21. 2024HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fieldscited by 14
  22. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 14
  23. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  24. 2023MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervisioncited by 12
  25. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  26. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  27. 2023Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Imagescited by 10
  28. 2023FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flowcited by 10
  29. 2023CHORD: Category-level Hand-held Object Reconstruction via Shape Deformationcited by 10
  30. 2023Dynamic Hyperbolic Attention Network for Fine Hand-object Reconstructioncited by 10
  31. 2023HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Imagecited by 10
  32. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  33. 2025BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splattingcited by 10
  34. 2025How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactionscited by 10
  35. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10
  36. 2024DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Imagecited by 9
  37. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9
  38. 2023DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulationcited by 8

50 most cited of 90 All 90 (JSON)