# Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction

arXiv: https://arxiv.org/abs/2512.00960 · HTML: https://mlsanity.com/paper/2512.00960/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 87 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Builds on (most cited first, 30 of 68)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction](https://mlsanity.com/paper/2004.13449/) (2020)

## Built on by (most cited first, 3 of 3)

- [LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model](https://mlsanity.com/paper/2603.27449/) (2026)
- [MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives](https://mlsanity.com/paper/2604.24833/) (2026)
- [Recovering Physically Plausible Human-Object Interactions from Monocular Videos](https://mlsanity.com/paper/2606.05359/) (2026)

## Abstract (arXiv)

Generalized robots must learn from diverse, large-scale human-object interactions (HOI) to operate robustly in the real world. Monocular internet videos offer a nearly limitless and readily available source of data, capturing an unparalleled diversity of human activities, objects, and environments. However, accurately and scalably extracting 4D interaction data from these in-the-wild videos remains a significant and unsolved challenge. To overcome the annotation bottleneck, we introduce an efficient sparse contact annotation paradigm. To scale this process, we develop InterPoint, a multi-modal predictor that drives a human-in-the-loop data engine. Building upon these efficiently acquired annotations, we introduce 4DHOISolver, a novel optimization framework that constrains the ill-posed 4D HOI reconstruction problem, maintaining high spatio-temporal coherence and physical plausibility. Leveraging this framework, we introduce Open4DHOI, a new large-scale 4D HOI dataset featuring a diverse catalog of 135 object types and 133 actions. Furthermore, we demonstrate the effectiveness of our reconstructions by enabling an RL-based agent to imitate the recovered motions. Data and code will be publicly available at https://github.com/wenboran2002/open4dhoi_code.