mlsanityData to 7 Oct 2026

Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips

arXiv 2309.05663 ↗First posted 11 Sep 2023cs.CVcited by 73

Not on the map. Why?

Abstract (arXiv)

We tackle the task of reconstructing hand-object interactions from short video clips. Given an input video, our approach casts 3D inference as a per-video optimization and recovers a neural 3D representation of the object shape, as well as the time-varying motion and hand articulation. While the input video naturally provides some multi-view cues to guide 3D inference, these are insufficient on their own due to occlusions and limited viewpoint variations. To obtain accurate 3D, we augment the multi-view signals with generic data-driven priors to guide reconstruction. Specifically, we learn a diffusion network to model the conditional distribution of (geometric) renderings of objects conditioned on hand configuration and category label, and leverage it as a prior to guide the novel-view renderings of the reconstructed scene. We empirically evaluate our approach on egocentric videos across 6 object categories, and observe significant improvements over prior single-view and multi-view methods. Finally, we demonstrate our system's ability to reconstruct arbitrary clips from YouTube, showing both 1st and 3rd person interactions.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 78 of its 91 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

78 of 91 references.

References · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  9. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  10. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  11. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
38 more
  1. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  3. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  4. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  6. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  7. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  8. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  9. 2019PointRend: Image Segmentation as Renderingcited by 439
  10. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  11. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  12. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  13. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  14. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  15. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  16. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  17. 2022Novel View Synthesis with Diffusion Modelscited by 299
  18. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  19. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  20. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  21. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  22. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  23. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  24. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  25. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
  26. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  27. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  28. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  29. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  30. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  31. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priorscited by 156
  32. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  33. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  34. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  35. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  36. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  37. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  38. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 24; 2025: 27; 2026: 19.

73 papers.

Sort:

By area:

  1. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  2. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  3. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  4. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  5. 2024GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusioncited by 33
  6. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  7. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  8. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  9. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  10. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 22
  11. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  12. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
38 more
  1. 2024HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awarenesscited by 21
  2. 2024Controlling the World by Sleight of Handcited by 19
  3. 2025Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Modelcited by 19
  4. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 19
  5. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  6. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 17
  7. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  8. 2025MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clipscited by 15
  9. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
  10. 2024HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactionscited by 13
  11. 2023MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervisioncited by 12
  12. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 12
  13. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 11
  14. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  15. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  16. 2024RHOBIN Challenge: Reconstruction of Human Object Interactioncited by 10
  17. 2024Physics-based Scene Layout Generation from Human Motioncited by 10
  18. 2024NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Modelcited by 10
  19. 2025BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splattingcited by 10
  20. 2025How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactionscited by 10
  21. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10
  22. 2024DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Imagecited by 9
  23. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  24. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9
  25. 2024Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representationcited by 8
  26. 2026Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstructioncited by 8
  27. 2025Zero-Shot Human-Object Interaction Synthesis with Multimodal Priorscited by 7
  28. 2025Novel Diffusion Models for Multimodal 3D Hand Trajectory Predictioncited by 7
  29. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  30. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  31. 2026OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generationcited by 7
  32. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 6
  33. 2024Sparse multi-view hand-object reconstruction for unseen environmentscited by 4
  34. 2024Joint Diffusion for Universal Hand-Object Grasp Generationcited by 4
  35. 2025SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectoriescited by 3
  36. 2025iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformercited by 3
  37. 2025HOSt3R: Keypoint-free Hand-Object 3D Reconstruction from RGB imagescited by 3
  38. 2024Stereo Hand-Object Reconstruction for Human-to-Robot Handovercited by 2

50 most cited of 73 All 73 (JSON)