# Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image

arXiv: https://arxiv.org/abs/2002.12212 · HTML: https://mlsanity.com/paper/2002.12212/

## Key facts

- Cited by 172 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2020.

## Built on by (most cited first, 30 of 172)

- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation](https://mlsanity.com/paper/2412.03558/) (2024)
- [Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks](https://mlsanity.com/paper/2108.07478/) (2021)
- [Holistic 3D Scene Understanding from a Single Image with Implicit Representation](https://mlsanity.com/paper/2103.06422/) (2021)
- [GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Voting](https://mlsanity.com/paper/2203.07918/) (2022)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [Visual Spatial Tuning](https://mlsanity.com/paper/2511.05491/) (2025)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Object-Compositional Neural Implicit Surfaces](https://mlsanity.com/paper/2207.09686/) (2022)
- [ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimization](https://mlsanity.com/paper/2207.13691/) (2022)
- [Panoptic 3D Scene Reconstruction From a Single RGB Image](https://mlsanity.com/paper/2111.02444/) (2021)
- [CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation](https://mlsanity.com/paper/2203.01929/) (2022)
- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single View](https://mlsanity.com/paper/2404.03421/) (2024)
- [Human-Aware Object Placement for Visual Environment Reconstruction](https://mlsanity.com/paper/2203.03609/) (2022)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields](https://mlsanity.com/paper/2212.02501/) (2022)
- [DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image](https://mlsanity.com/paper/2311.18610/) (2023)
- [PhyRecon: Physically Plausible Neural Scene Reconstruction](https://mlsanity.com/paper/2404.16666/) (2024)
- [Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes](https://mlsanity.com/paper/2207.08656/) (2022)
- [RBP-Pose: Residual Bounding Box Projection for Category-Level Pose Estimation](https://mlsanity.com/paper/2208.00237/) (2022)
- [IST-Net: Prior-free Category-level Pose Estimation with Implicit Space Transformation](https://mlsanity.com/paper/2303.13479/) (2023)

## Abstract (arXiv)

Semantic reconstruction of indoor scenes refers to both scene understanding and object reconstruction. Existing works either address one part of this problem or focus on independent objects. In this paper, we bridge the gap between understanding and reconstruction, and propose an end-to-end solution to jointly reconstruct room layout, object bounding boxes and meshes from a single image. Instead of separately resolving scene understanding and object reconstruction, our method builds upon a holistic scene context and proposes a coarse-to-fine hierarchy with three components: 1. room layout with camera pose; 2. 3D object bounding boxes; 3. object meshes. We argue that understanding the context of each component can assist the task of parsing the others, which enables joint understanding and reconstruction. The experiments on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently outperforms existing methods in indoor layout estimation, 3D object detection and mesh reconstruction.