# Zero-shot Reconstruction of In-Scene Object Manipulation from Video

arXiv: https://arxiv.org/abs/2512.19684 · HTML: https://mlsanity.com/paper/2512.19684/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2025.

## Built on by (most cited first, 2 of 2)

- [4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction](https://mlsanity.com/paper/2610.08782/) (2026)
- [Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics](https://mlsanity.com/paper/2610.15769/) (2026)

## Abstract (arXiv)

We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video. It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions. Existing methods operate in hand centric coordinates and ignore the scene, hindering metric accuracy and practical use. In our method, we first use data-driven foundation models to initialize the core components, including the object mesh and poses, the scene point cloud, and the hand poses. We then apply a two-stage optimization that recovers a complete hand-object motion from grasping to interaction, which remains consistent with the scene information observed in the input video.