# Reconstructing Hand-Held Objects from Monocular Video

arXiv: https://arxiv.org/abs/2211.16835 · HTML: https://mlsanity.com/paper/2211.16835/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 74 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 30 of 50)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Neural Unsigned Distance Fields for Implicit Function Learning](https://mlsanity.com/paper/2010.13938/) (2020)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)

## Built on by (most cited first, 30 of 35)

- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis](https://mlsanity.com/paper/2404.12383/) (2024)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects](https://mlsanity.com/paper/2403.16428/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips](https://mlsanity.com/paper/2508.05506/) (2025)
- [SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction](https://mlsanity.com/paper/2309.10748/) (2023)
- [3D Reconstruction of Objects in Hands without Real World 3D Supervision](https://mlsanity.com/paper/2305.03036/) (2023)
- [MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision](https://mlsanity.com/paper/2310.11696/) (2023)
- [HORT: Monocular Hand-held Objects Reconstruction with Transformers](https://mlsanity.com/paper/2503.21313/) (2025)
- [Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videos](https://mlsanity.com/paper/2312.15719/) (2023)
- [How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions](https://mlsanity.com/paper/2504.12284/) (2025)
- [WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos](https://mlsanity.com/paper/2602.22209/) (2026)
- [Color-NeuS: Reconstructing Neural Implicit Surfaces with Color](https://mlsanity.com/paper/2308.06962/) (2023)
- [Ins-HOI: Instance Aware Human-Object Interactions Recovery](https://mlsanity.com/paper/2312.09641/) (2023)
- [ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos](https://mlsanity.com/paper/2602.06226/) (2026)
- [UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation](https://mlsanity.com/paper/2503.13303/) (2025)
- [Choreographing a World of Dynamic Objects](https://mlsanity.com/paper/2601.04194/) (2026)
- [In-Hand 3D Object Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2312.16425/) (2023)
- [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026)
- [Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera](https://mlsanity.com/paper/2405.05858/) (2024)
- [Benchmarking 2D Egocentric Hand Pose Datasets](https://mlsanity.com/paper/2409.07337/) (2024)
- [Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery](https://mlsanity.com/paper/2505.17677/) (2025)
- [HOSt3R: Keypoint-free Hand-Object 3D Reconstruction from RGB images](https://mlsanity.com/paper/2508.16465/) (2025)
- [Reconstructing Objects along Hand Interaction Timelines in Egocentric Video](https://mlsanity.com/paper/2512.07394/) (2025)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects](https://mlsanity.com/paper/2502.13968/) (2025)

## Abstract (arXiv)

This paper presents an approach that reconstructs a hand-held object from a monocular video. In contrast to many recent methods that directly predict object geometry by a trained network, the proposed approach does not require any learned prior about the object and is able to recover more accurate and detailed object geometry. The key idea is that the hand motion naturally provides multiple views of the object and the motion can be reliably estimated by a hand pose tracker. Then, the object geometry can be recovered by solving a multi-view reconstruction problem. We devise an implicit neural representation-based method to solve the reconstruction problem and address the issues of imprecise hand pose estimation, relative hand-object motion, and insufficient geometry optimization for small objects. We also provide a newly collected dataset with 3D ground truth to validate the proposed approach.