# 3D Object Reconstruction from Hand-Object Interactions

arXiv: https://arxiv.org/abs/1704.00529 · HTML: https://mlsanity.com/paper/1704.00529/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2017.

## Built on by (most cited first, 30 of 31)

- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation](https://mlsanity.com/paper/2104.14639/) (2021)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [Towards unconstrained joint hand-object reconstruction from RGB videos](https://mlsanity.com/paper/2108.07044/) (2021)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [Reconstructing Hand-Held Objects from Monocular Video](https://mlsanity.com/paper/2211.16835/) (2022)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [3D Hand Pose Estimation in Everyday Egocentric Images](https://mlsanity.com/paper/2312.06583/) (2023)
- [In-Hand 3D Object Scanning from an RGB Sequence](https://mlsanity.com/paper/2211.16193/) (2022)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [Visual-Tactile Sensing for In-Hand Object Reconstruction](https://mlsanity.com/paper/2303.14498/) (2023)
- [HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects](https://mlsanity.com/paper/2301.07652/) (2023)
- [3D Reconstruction of Objects in Hands without Real World 3D Supervision](https://mlsanity.com/paper/2305.03036/) (2023)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [Decomposed Vector-Quantized Variational Autoencoder for Human Grasp Generation](https://mlsanity.com/paper/2407.14062/) (2024)
- [D-SCo: Dual-Stream Conditional Diffusion for Monocular Hand-Held Object Reconstruction](https://mlsanity.com/paper/2311.14189/) (2023)
- [Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera](https://mlsanity.com/paper/2405.05858/) (2024)
- [Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction](https://mlsanity.com/paper/2110.10174/) (2021)
- [ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction from a Single Depth Map](https://mlsanity.com/paper/2310.11811/) (2023)
- [Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video](https://mlsanity.com/paper/2308.04074/) (2023)
- [PickScan: Object discovery and reconstruction from handheld interactions](https://mlsanity.com/paper/2411.11196/) (2024)
- [PartHOI: Part-based Hand-Object Interaction Transfer via Generalized Cylinders](https://mlsanity.com/paper/2504.20599/) (2025)
- [Paparazzo: Active Mapping of Moving 3D Objects](https://mlsanity.com/paper/2604.19556/) (2026)

## Abstract (arXiv)

Recent advances have enabled 3d object reconstruction approaches using a single off-the-shelf RGB-D camera. Although these approaches are successful for a wide range of object classes, they rely on stable and distinctive geometric or texture features. Many objects like mechanical parts, toys, household or decorative articles, however, are textureless and characterized by minimalistic shapes that are simple and symmetric. Existing in-hand scanning systems and 3d reconstruction techniques fail for such symmetric objects in the absence of highly distinctive features. In this work, we show that extracting 3d hand motion for in-hand scanning effectively facilitates the reconstruction of even featureless and highly symmetric objects and we present an approach that fuses the rich additional information of hands into a 3d reconstruction pipeline, significantly contributing to the state-of-the-art of in-hand scanning.