mlsanityData to 14 Oct 2026

HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image

arXiv 2309.07891 ↗First posted 14 Sep 2023cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper presents a method to learn hand-object interaction prior for reconstructing a 3D hand-object scene from a single RGB image. The inference as well as training-data generation for 3D hand-object scene reconstruction is challenging due to the depth ambiguity of a single image and occlusions by the hand and object. We turn this challenge into an opportunity by utilizing the hand shape to constrain the possible relative configuration of the hand and object geometry. We design a generalizable implicit function, HandNeRF, that explicitly encodes the correlation of the 3D hand shape features and 2D object features to predict the hand and object scene geometry. With experiments on real-world datasets, we show that HandNeRF is able to reconstruct hand-object scenes of novel grasp configurations more accurately than comparable methods. Moreover, we demonstrate that object reconstruction from HandNeRF ensures more accurate execution of downstream tasks, such as grasping and motion planning for robotic hand-over and manipulation. Homepage: https://samsunglabs.github.io/HandNeRF-project-page/

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

45 of 54 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2023Segment Anythingcited by 8,234
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  9. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  10. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  12. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
33 more
  1. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  2. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  3. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  4. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  5. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  6. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  7. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  8. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  9. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  10. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  11. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  12. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  13. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  14. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 218
  15. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  16. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  17. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  18. 2021Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Renderingcited by 158
  19. 2021H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motioncited by 157
  20. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  21. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  22. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  23. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  24. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  25. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  26. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 96
  27. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  28. 2020Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimationcited by 81
  29. 2016Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Inputcited by 56
  30. 2021Contrastive Representation Learning for Hand Shape Estimationcited by 34
  31. 2021HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Datasetcited by 26
  32. 2022MonoNHR: Monocular Neural Human Renderercited by 15
  33. 2022RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Controlcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 2.

10 papers.

  1. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  2. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  3. 2025UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimationcited by 6
  4. 2024Sparse multi-view hand-object reconstruction for unseen environmentscited by 4
  5. 2024Stereo Hand-Object Reconstruction for Human-to-Robot Handovercited by 2
  6. 2025Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Controlcited by 2
  7. 2025TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenescited by 1
  8. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1
  9. 2025Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priorscited by 0
  10. 2026MASS: Mesh-inellipse Aligned Deformable Surfel Splatting for Hand Reconstruction and Rendering from Egocentric Monocular Videocited by 0