# SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans

arXiv: https://arxiv.org/abs/2003.12622 · HTML: https://mlsanity.com/paper/2003.12622/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2020.

## Built on by (most cited first, 30 of 44)

- [SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model](https://mlsanity.com/paper/2403.13064/) (2024)
- [Holistic 3D Scene Understanding from a Single Image with Implicit Representation](https://mlsanity.com/paper/2103.06422/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [SpatialLM: Training Large Language Models for Structured Indoor Modeling](https://mlsanity.com/paper/2506.07491/) (2025)
- [Semantic Scene Completion via Integrating Instances and Scene in-the-Loop](https://mlsanity.com/paper/2104.03640/) (2021)
- [Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries](https://mlsanity.com/paper/2211.15658/) (2022)
- [DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image](https://mlsanity.com/paper/2311.18610/) (2023)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization](https://mlsanity.com/paper/2108.10743/) (2021)
- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Clouds](https://mlsanity.com/paper/2109.05566/) (2021)
- [PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes](https://mlsanity.com/paper/2207.00757/) (2022)
- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [Monte Carlo Scene Search for 3D Scene Understanding](https://mlsanity.com/paper/2103.07969/) (2021)
- [SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB image](https://mlsanity.com/paper/2210.01044/) (2022)
- [From Semi-supervised to Omni-supervised Room Layout Estimation Using Point Clouds](https://mlsanity.com/paper/2301.13865/) (2023)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)
- [PolyRoom: Room-aware Transformer for Floorplan Reconstruction](https://mlsanity.com/paper/2407.10439/) (2024)
- [Multi-instance Point Cloud Registration by Efficient Correspondence Clustering](https://mlsanity.com/paper/2111.14582/) (2021)
- [Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects](https://mlsanity.com/paper/2205.02714/) (2022)
- [FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos](https://mlsanity.com/paper/2403.15161/) (2024)
- [Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes](https://mlsanity.com/paper/2404.04557/) (2024)
- [Estimating Generic 3D Room Structures from 2D Annotations](https://mlsanity.com/paper/2306.09077/) (2023)
- [FRI-Net: Floorplan Reconstruction via Room-wise Implicit Representation](https://mlsanity.com/paper/2407.10687/) (2024)
- [REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image](https://mlsanity.com/paper/2605.30338/) (2026)
- [MCTS with Refinement for Proposals Selection Games in Scene Understanding](https://mlsanity.com/paper/2207.03204/) (2022)
- [Sparse Multi-Object Render-and-Compare](https://mlsanity.com/paper/2310.11184/) (2023)
- [CAGE: Continuity-Aware edGE Network Unlocks Robust Floorplan Reconstruction](https://mlsanity.com/paper/2509.15459/) (2025)

## Abstract (arXiv)

We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors. Our key idea is to jointly optimize for both CAD model alignments as well as layout estimations of the scanned scene, explicitly modeling inter-relationships between objects-to-objects and objects-to-layout. Since object arrangement and scene layout are intrinsically coupled, we show that treating the problem jointly significantly helps to produce globally-consistent representations of a scene. Object CAD models are aligned to the scene by establishing dense correspondences between geometry, and we introduce a hierarchical layout prediction approach to estimate layout planes from corners and edges of the scene.To this end, we propose a message-passing graph neural network to model the inter-relationships between objects and layout, guiding generation of a globally object alignment in a scene. By considering the global scene layout, we achieve significantly improved CAD alignments compared to state-of-the-art methods, improving from 41.83% to 58.41% alignment accuracy on SUNCG and from 50.05% to 61.24% on ScanNet, respectively. The resulting CAD-based representations makes our method well-suited for applications in content creation such as augmented- or virtual reality.