# Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

arXiv: https://arxiv.org/abs/2608.27407 · HTML: https://mlsanity.com/paper/2608.27407/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 70 of its 80 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 30 of 70)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion](https://mlsanity.com/paper/2403.12008/) (2024)
- [Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation](https://mlsanity.com/paper/2501.12202/) (2025)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)

## Abstract (arXiv)

Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.