mlsanityData to 14 Oct 2026

HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video

arXiv 2311.18448 ↗First posted 30 Nov 2023cs.CVcited by 78

Not on the map. Why?

Abstract (arXiv)

Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour. However, most existing methods for hand-object reconstruction from RGB either assume pre-scanned object templates or heavily rely on limited 3D hand-object data, restricting their ability to scale and generalize to more unconstrained interaction settings. To this end, we introduce HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can reconstruct disentangled 3D hand and object from 2D images. We also further incorporate hand-object constraints to improve hand-object poses and consequently the reconstruction quality. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we qualitatively show its robustness in reconstructing from in-the-wild videos. Code: https://github.com/zc-alexfan/hold

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 74 of its 93 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

74 of 93 references.

References · 74

  1. 2023Segment Anythingcited by 8,234
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  3. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  4. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  5. 2021Volume Rendering of Neural Implicit Surfacescited by 996
  6. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  7. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  8. 2020Implicit Geometric Regularization for Learning Shapescited by 710
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
  10. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 559
  11. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 495
  12. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
38 more
  1. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  2. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  3. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  4. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  5. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 260
  6. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  7. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  8. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  9. 2021SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapescited by 221
  10. 2023Segment and Track Anythingcited by 193
  11. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  12. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 181
  13. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  14. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 161
  15. 2023HUGS: Human Gaussian Splatscited by 154
  16. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  17. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 148
  18. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  19. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  20. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  21. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  22. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  23. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  24. 2023Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decompositioncited by 134
  25. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  26. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  27. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 113
  28. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  29. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  30. 2020Category Level Object Pose Estimation via Neural Analysis-by-Synthesiscited by 100
  31. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  32. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  33. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  34. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  35. 2023Hi4D: 4D Instance Segmentation of Close Human Interactioncited by 87
  36. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  37. 2023Detector-Free Structure from Motioncited by 71
  38. 2016Amodal Instance Segmentationcited by 63

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 28; 2026: 35.

78 papers.

Sort:

By area:

  1. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  2. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 60
  3. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  4. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 36
  5. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  6. 2025SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrationscited by 25
  7. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 23
  8. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 23
  9. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Modelscited by 21
  10. 2025MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clipscited by 17
  11. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  12. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
38 more
  1. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  2. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  3. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 11
  4. 2026Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstructioncited by 11
  5. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  6. 2025BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splattingcited by 10
  7. 2024DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Imagecited by 9
  8. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  9. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  10. 2024Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representationcited by 8
  11. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 8
  12. 2025Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidancecited by 8
  13. 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactionscited by 8
  14. 2026WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videoscited by 8
  15. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 7
  16. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  17. 20243D Whole-body Grasp Synthesis with Directional Controllabilitycited by 6
  18. 2026Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Modelscited by 5
  19. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  20. 2026AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generationcited by 4
  21. 2026ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactionscited by 4
  22. 2026HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesiscited by 4
  23. 2024ContactHandover: Contact-Guided Robot-to-Human Object Handovercited by 3
  24. 2024Free-Moving Object Reconstruction and Pose Estimation with Virtual Cameracited by 3
  25. 2024Benchmarking 2D Egocentric Hand Pose Datasetscited by 3
  26. 2025QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objectscited by 3
  27. 2025SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectoriescited by 3
  28. 2025HOSt3R: Keypoint-free Hand-Object 3D Reconstruction from RGB imagescited by 3
  29. 2025PALM: A Dataset and Baseline for Learning Multi-subject Hand Priorcited by 3
  30. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 3
  31. 2026GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splattingcited by 3
  32. 2026Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactionscited by 3
  33. 2026Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstructioncited by 3
  34. 2025FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidancecited by 2
  35. 2026HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactionscited by 2
  36. 2026HandX: Scaling Bimanual Motion and Interaction Generationcited by 2
  37. 2026VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulationcited by 2
  38. 2026Towards in-the-wild Egocentric 3D Hand-Object Pose Estimationcited by 2

50 most cited of 78 All 78 (JSON)