# In-Hand 3D Object Scanning from an RGB Sequence

arXiv: https://arxiv.org/abs/2211.16193 · HTML: https://mlsanity.com/paper/2211.16193/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections](https://mlsanity.com/paper/2205.15768/) (2022)
- [Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era](https://mlsanity.com/paper/1906.06543/) (2019)
- [FroDO: From Detections to 3D Objects](https://mlsanity.com/paper/2005.05125/) (2020)
- [NeROIC: Neural Rendering of Objects from Online Image Collections](https://mlsanity.com/paper/2201.02533/) (2022)
- [3D Object Reconstruction from Hand-Object Interactions](https://mlsanity.com/paper/1704.00529/) (2017)
- [DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration](https://mlsanity.com/paper/2112.02849/) (2021)
- [Measuring robustness of Visual SLAM](https://mlsanity.com/paper/1910.04755/) (2019)
- ["What's This?" -- Learning to Segment Unknown Objects from Manipulation Sequences](https://mlsanity.com/paper/2011.03279/) (2020)

## Built on by (most cited first, 27 of 27)

- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking](https://mlsanity.com/paper/2406.09598/) (2024)
- [G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis](https://mlsanity.com/paper/2404.12383/) (2024)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimation](https://mlsanity.com/paper/2504.02812/) (2025)
- [MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips](https://mlsanity.com/paper/2508.05506/) (2025)
- [HORT: Monocular Hand-held Objects Reconstruction with Transformers](https://mlsanity.com/paper/2503.21313/) (2025)
- [Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videos](https://mlsanity.com/paper/2312.15719/) (2023)
- [RHOBIN Challenge: Reconstruction of Human Object Interaction](https://mlsanity.com/paper/2401.04143/) (2024)
- [Color-NeuS: Reconstructing Neural Implicit Surfaces with Color](https://mlsanity.com/paper/2308.06962/) (2023)
- [Tracking by 3D Model Estimation of Unknown Objects in Videos](https://mlsanity.com/paper/2304.06419/) (2023)
- [In-Hand 3D Object Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2312.16425/) (2023)
- [Sparse multi-view hand-object reconstruction for unseen environments](https://mlsanity.com/paper/2405.01353/) (2024)
- [Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera](https://mlsanity.com/paper/2405.05858/) (2024)
- [Reconstructing Objects along Hand Interaction Timelines in Egocentric Video](https://mlsanity.com/paper/2512.07394/) (2025)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [Stereo Hand-Object Reconstruction for Human-to-Robot Handover](https://mlsanity.com/paper/2412.07487/) (2024)
- [OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects](https://mlsanity.com/paper/2510.20605/) (2025)
- [Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation](https://mlsanity.com/paper/2606.30598/) (2026)
- [Grasp-and-Lift: Executable 3D Hand-Object Interaction Reconstruction via Physics-in-the-Loop Optimization](https://mlsanity.com/paper/2601.18121/) (2026)
- [PickScan: Object discovery and reconstruction from handheld interactions](https://mlsanity.com/paper/2411.11196/) (2024)
- [Placeit! A Framework for Learning Robot Object Placement Skills](https://mlsanity.com/paper/2510.09267/) (2025)
- [Paparazzo: Active Mapping of Moving 3D Objects](https://mlsanity.com/paper/2604.19556/) (2026)

## Abstract (arXiv)

We propose a method for in-hand 3D scanning of an unknown object with a monocular camera. Our method relies on a neural implicit surface representation that captures both the geometry and the appearance of the object, however, by contrast with most NeRF-based methods, we do not assume that the camera-object relative poses are known. Instead, we simultaneously optimize both the object shape and the pose trajectory. As direct optimization over all shape and pose parameters is prone to fail without coarse-level initialization, we propose an incremental approach that starts by splitting the sequence into carefully selected overlapping segments within which the optimization is likely to succeed. We reconstruct the object shape and track its poses independently within each segment, then merge all the segments before performing a global optimization. We show that our method is able to reconstruct the shape and color of both textured and challenging texture-less objects, outperforms classical methods that rely only on appearance features, and that its performance is close to recent methods that assume known camera poses.