# MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips arXiv: https://arxiv.org/abs/2508.05506 ยท HTML: https://mlsanity.com/paper/2508.05506/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. - Builds on 80 of its 95 references that are in the map. - First posted on arXiv 7 Aug 2025. ## Placed in - Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/) ## Builds on (most cited first, 30 of 80) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022) - [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023) - [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019) - [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022) - [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022) - [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021) - [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024) - [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023) - [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023) - [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023) - [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024) - [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024) - [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023) - [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023) - [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018) - [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024) - [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019) - [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022) - [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023) - [LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching](https://mlsanity.com/paper/2311.11284/) (2023) - [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019) - [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020) - [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017) - [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023) - [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017) - [CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model](https://mlsanity.com/paper/2403.05034/) (2024) - [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019) ## Built on by (most cited first, 17 of 17) - [ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos](https://mlsanity.com/paper/2602.06226/) (2026) - [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025) - [AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation](https://mlsanity.com/paper/2602.04672/) (2026) - [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026) - [HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2607.17097/) (2026) - [PALM: A Dataset and Baseline for Learning Multi-subject Hand Prior](https://mlsanity.com/paper/2511.05403/) (2025) - [Reconstructing Objects along Hand Interaction Timelines in Egocentric Video](https://mlsanity.com/paper/2512.07394/) (2025) - [PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generation](https://mlsanity.com/paper/2603.22193/) (2026) - [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026) - [Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction](https://mlsanity.com/paper/2605.20992/) (2026) - [Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction](https://mlsanity.com/paper/2511.14540/) (2025) - [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026) - [High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians](https://mlsanity.com/paper/2606.15908/) (2026) - [C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video](https://mlsanity.com/paper/2608.07045/) (2026) - [DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction](https://mlsanity.com/paper/2609.36454/) (2026) - [Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics](https://mlsanity.com/paper/2610.15769/) (2026) - [DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation](https://mlsanity.com/paper/2610.19089/) (2026) ## Abstract (arXiv) Most RGB-based hand-object reconstruction methods rely on object templates, while template-free methods typically assume full object visibility. This assumption often breaks in real-world settings, where fixed camera viewpoints and static grips leave parts of the object unobserved, resulting in implausible reconstructions. To overcome this, we present MagicHOI, a method for reconstructing hands and objects from short monocular interaction videos, even under limited viewpoint variation. Our key insight is that, despite the scarcity of paired 3D hand-object data, large-scale novel view synthesis diffusion models offer rich object supervision. This supervision serves as a prior to regularize unseen object regions during hand interactions. Leveraging this insight, we integrate a novel view synthesis model into our hand-object reconstruction framework. We further align hand to object by incorporating visible contact constraints. Our results demonstrate that MagicHOI significantly outperforms existing state-of-the-art hand-object reconstruction methods. We also show that novel view synthesis diffusion priors effectively regularize unseen object regions, enhancing 3D hand-object reconstruction.