mlsanityData to 14 Oct 2026

Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views

arXiv 2308.11198 ↗First posted 22 Aug 2023cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Hand-object interaction understanding and the barely addressed novel view synthesis are highly desired in the immersive communication, whereas it is challenging due to the high deformation of hand and heavy occlusions between hand and object. In this paper, we propose a neural rendering and pose estimation system for hand-object interaction from sparse views, which can also enable 3D hand-object interaction editing. We share the inspiration from recent scene understanding work that shows a scene specific model built beforehand can significantly improve and unblock vision tasks especially when inputs are sparse, and extend it to the dynamic hand-object interaction scenario and propose to solve the problem in two stages. We first learn the shape and appearance prior knowledge of hands and objects separately with the neural representation at the offline stage. During the online stage, we design a rendering-based joint model fitting framework to understand the dynamic hand-object interaction with the pre-built hand and object models as well as interaction priors, which thereby overcomes penetration and separation issues between hand and object and also enables novel view synthesis. In order to get stable contact during the hand-object interaction process in a sequence, we propose a stable contact loss to make the contact region to be consistent. Experiments demonstrate that our method outperforms the state-of-the-art methods. Code and dataset are available in project webpage https://iscas3dv.github.io/HO-NeRF.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 57 of its 72 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

57 of 72 references.

References · 57

  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  2. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  3. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  4. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  5. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  7. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  8. 2021Volume Rendering of Neural Implicit Surfacescited by 996
  9. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  10. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  11. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  12. 2020Implicit Geometric Regularization for Learning Shapescited by 710
38 more
  1. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  2. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  3. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  4. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  5. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  6. 2019PointRend: Image Segmentation as Renderingcited by 440
  7. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  8. 2021Animatable Neural Radiance Fields for Modeling Dynamic Human Bodiescited by 361
  9. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  10. 2018Learning 3D Human Dynamics from Videocited by 316
  11. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  12. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  13. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  14. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  15. 2021A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Posecited by 233
  16. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  17. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  18. 2021SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapescited by 221
  19. 2021Neural Articulated Radiance Fieldcited by 196
  20. 2021InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Renderingcited by 182
  21. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  22. 2019Learnable Triangulation of Human Posecited by 161
  23. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  24. 2019NASA: Neural Articulated Shape Approximationcited by 149
  25. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  26. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  27. 2022ARAH: Animatable Volume Rendering of Articulated Human SDFscited by 139
  28. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  29. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  30. 2020Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondencecited by 119
  31. 2022Structured Local Radiance Fields for Human Avatar Modelingcited by 116
  32. 2021LEAP: Learning Articulated Occupancy of Peoplecited by 107
  33. 2020Object-Centric Neural Scene Renderingcited by 90
  34. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  35. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 82
  36. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 78
  37. 2022KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypointscited by 77
  38. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 4; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
  2. 2024GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusioncited by 35
  3. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  4. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  5. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 23
  6. 2024OHTA: One-shot Hand Avatar via Data-driven Implicit Priorscited by 18
  7. 2025MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clipscited by 17
  8. 2024Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamicscited by 15
  9. 2025Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidancecited by 8
  10. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 7
  11. 2024Sparse multi-view hand-object reconstruction for unseen environmentscited by 4
  12. 2025HOSt3R: Keypoint-free Hand-Object 3D Reconstruction from RGB imagescited by 3
4 more
  1. 2025MultiModal Action Conditioned Video Generationcited by 3
  2. 2024Stereo Hand-Object Reconstruction for Human-to-Robot Handovercited by 2
  3. 2026DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstructioncited by 1
  4. 2026NVS-HO: A Benchmark for Novel View Synthesis of Handheld Objectscited by 0