# Monocular Human-Object Reconstruction in the Wild

arXiv: https://arxiv.org/abs/2407.20566 · HTML: https://mlsanity.com/paper/2407.20566/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023)
- [Probabilistic Modeling for Human Mesh Recovery](https://mlsanity.com/paper/2108.11944/) (2021)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation](https://mlsanity.com/paper/2004.03686/) (2020)
- [Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks](https://mlsanity.com/paper/1909.00321/) (2019)
- [Unsupervised 3D Pose Estimation with Geometric Self-Supervision](https://mlsanity.com/paper/1904.04812/) (2019)
- [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023)
- [BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training](https://mlsanity.com/paper/2203.13249/) (2022)
- [Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors](https://mlsanity.com/paper/2209.02485/) (2022)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [Learning 3D Object Shape and Layout without 3D Supervision](https://mlsanity.com/paper/2206.07028/) (2022)
- [CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images](https://mlsanity.com/paper/2308.12288/) (2023)
- [Diffusion Models are Efficient Data Generators for Human Mesh Recovery](https://mlsanity.com/paper/2403.11111/) (2024)
- [Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wild](https://mlsanity.com/paper/2403.14539/) (2024)

## Built on by (most cited first, 9 of 9)

- [OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model](https://mlsanity.com/paper/2505.18947/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction](https://mlsanity.com/paper/2512.11988/) (2025)
- [Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions](https://mlsanity.com/paper/2503.15898/) (2025)
- [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025)
- [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026)
- [Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction](https://mlsanity.com/paper/2512.00960/) (2025)
- [End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction](https://mlsanity.com/paper/2603.14435/) (2026)
- [Recovering Physically Plausible Human-Object Interactions from Monocular Videos](https://mlsanity.com/paper/2606.05359/) (2026)

## Abstract (arXiv)

Learning the prior knowledge of the 3D human-object spatial relation is crucial for reconstructing human-object interaction from images and understanding how humans interact with objects in 3D space. Previous works learn this prior from datasets collected in controlled environments, but due to the diversity of domains, they struggle to generalize to real-world scenarios. To overcome this limitation, we present a 2D-supervised method that learns the 3D human-object spatial relation prior purely from 2D images in the wild. Our method utilizes a flow-based neural network to learn the prior distribution of the 2D human-object keypoint layout and viewports for each image in the dataset. The effectiveness of the prior learned from 2D images is demonstrated on the human-object reconstruction task by applying the prior to tune the relative pose between the human and the object during the post-optimization stage. To validate and benchmark our method on in-the-wild images, we collect the WildHOI dataset from the YouTube website, which consists of various interactions with 8 objects in real-world scenarios. We conduct the experiments on the indoor BEHAVE dataset and the outdoor WildHOI dataset. The results show that our method achieves almost comparable performance with fully 3D supervised methods on the BEHAVE dataset, even if we have only utilized the 2D layout information, and outperforms previous methods in terms of generality and interaction diversity on in-the-wild images.