# Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

arXiv: https://arxiv.org/abs/1712.01812 · HTML: https://mlsanity.com/paper/1712.01812/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2017.

## Built on by (most cited first, 30 of 50)

- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images](https://mlsanity.com/paper/2208.07227/) (2022)
- [Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single View](https://mlsanity.com/paper/2404.03421/) (2024)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [Few-View Object Reconstruction with Unknown Categories and Camera Poses](https://mlsanity.com/paper/2212.04492/) (2022)
- [DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models](https://mlsanity.com/paper/2211.12131/) (2022)
- [Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes](https://mlsanity.com/paper/2207.08656/) (2022)
- [Planar Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2103.14644/) (2021)
- [D3D-HOI: Dynamic 3D Human-Object Interactions from Videos](https://mlsanity.com/paper/2108.08420/) (2021)
- [SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data](https://mlsanity.com/paper/2105.08612/) (2021)
- [Understanding 3D Object Articulation in Internet Videos](https://mlsanity.com/paper/2203.16531/) (2022)
- [Dynamic 3D Gaussian Fields for Urban Areas](https://mlsanity.com/paper/2406.03175/) (2024)
- [Learning to Imitate Object Interactions from Internet Videos](https://mlsanity.com/paper/2211.13225/) (2022)
- [Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images](https://mlsanity.com/paper/2108.09022/) (2021)
- [Multi-Level Neural Scene Graphs for Dynamic Urban Environments](https://mlsanity.com/paper/2404.00168/) (2024)
- [3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features](https://mlsanity.com/paper/2311.04391/) (2023)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [Monte Carlo Scene Search for 3D Scene Understanding](https://mlsanity.com/paper/2103.07969/) (2021)
- [CoCoNets: Continuous Contrastive 3D Scene Representations](https://mlsanity.com/paper/2104.03851/) (2021)
- [Learning 3D Object Shape and Layout without 3D Supervision](https://mlsanity.com/paper/2206.07028/) (2022)
- [PanoContext-Former: Panoramic Total Scene Understanding with a Transformer](https://mlsanity.com/paper/2305.12497/) (2023)
- [Structure from Silence: Learning Scene Structure from Ambient Sound](https://mlsanity.com/paper/2111.05846/) (2021)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds](https://mlsanity.com/paper/2204.09138/) (2022)
- [Gaussian Splatting in Style](https://mlsanity.com/paper/2403.08498/) (2024)
- [im2nerf: Image to Neural Radiance Field in the Wild](https://mlsanity.com/paper/2209.04061/) (2022)
- [External Knowledge Enhanced 3D Scene Generation from Sketch](https://mlsanity.com/paper/2403.14121/) (2024)

## Abstract (arXiv)

The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose. We propose a convolutional neural network-based approach to predict this representation and benchmark it on a large dataset of indoor scenes. Our experiments evaluate a number of practical design questions, demonstrate that we can infer this representation, and quantitatively and qualitatively demonstrate its merits compared to alternate representations.