# Sparse multi-view hand-object reconstruction for unseen environments

arXiv: https://arxiv.org/abs/2405.01353 · HTML: https://mlsanity.com/paper/2405.01353/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution](https://mlsanity.com/paper/2204.13062/) (2022)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion](https://mlsanity.com/paper/2006.15561/) (2020)
- [In-Hand 3D Object Scanning from an RGB Sequence](https://mlsanity.com/paper/2211.16193/) (2022)
- [Reactive Human-to-Robot Handovers of Arbitrary Objects](https://mlsanity.com/paper/2011.08961/) (2020)
- [Interacting Hand-Object Pose Estimation via Dense Mutual Attention](https://mlsanity.com/paper/2211.08805/) (2022)
- [Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision](https://mlsanity.com/paper/2006.01797/) (2020)
- [Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views](https://mlsanity.com/paper/2308.11198/) (2023)
- [SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction](https://mlsanity.com/paper/2309.10748/) (2023)
- [Towards safe human-to-robot handovers of unknown containers](https://mlsanity.com/paper/2107.01309/) (2021)
- [HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image](https://mlsanity.com/paper/2309.07891/) (2023)

## Built on by (most cited first, 4 of 4)

- [Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery](https://mlsanity.com/paper/2505.17677/) (2025)
- [Stereo Hand-Object Reconstruction for Human-to-Robot Handover](https://mlsanity.com/paper/2412.07487/) (2024)
- [Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splatting](https://mlsanity.com/paper/2503.09640/) (2025)
- [VEPHand: View-Efficient Photometric Hand Performance Capture at Scale](https://mlsanity.com/paper/2606.15966/) (2026)

## Abstract (arXiv)

Recent works in hand-object reconstruction mainly focus on the single-view and dense multi-view settings. On the one hand, single-view methods can leverage learned shape priors to generalise to unseen objects but are prone to inaccuracies due to occlusions. On the other hand, dense multi-view methods are very accurate but cannot easily adapt to unseen objects without further data collection. In contrast, sparse multi-view methods can take advantage of the additional views to tackle occlusion, while keeping the computational cost low compared to dense multi-view methods. In this paper, we consider the problem of hand-object reconstruction with unseen objects in the sparse multi-view setting. Given multiple RGB images of the hand and object captured at the same time, our model SVHO combines the predictions from each view into a unified reconstruction without optimisation across views. We train our model on a synthetic hand-object dataset and evaluate directly on a real world recorded hand-object dataset with unseen objects. We show that while reconstruction of unseen hands and objects from RGB is challenging, additional views can help improve the reconstruction quality.