# Monte Carlo Scene Search for 3D Scene Understanding

arXiv: https://arxiv.org/abs/2103.07969 · HTML: https://mlsanity.com/paper/2103.07969/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 63 references that are in the map.
- First posted on arXiv 14 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes](https://mlsanity.com/paper/2001.10692/) (2020)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation](https://mlsanity.com/paper/1901.03861/) (2019)
- [LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image](https://mlsanity.com/paper/1803.08999/) (2018)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense](https://mlsanity.com/paper/1909.01507/) (2019)
- [Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1808.02201/) (2018)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation](https://mlsanity.com/paper/1810.13049/) (2018)
- [Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene](https://mlsanity.com/paper/1712.01812/) (2017)
- [SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans](https://mlsanity.com/paper/2003.12622/) (2020)
- [End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans](https://mlsanity.com/paper/1906.04201/) (2019)
- [RoomNet: End-to-End Room Layout Estimation](https://mlsanity.com/paper/1703.06241/) (2017)
- [FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans](https://mlsanity.com/paper/1804.00090/) (2018)
- [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild](https://mlsanity.com/paper/1803.11493/) (2018)
- [3D-RelNet: Joint Object and Relational Network for 3D Prediction](https://mlsanity.com/paper/1906.02729/) (2019)
- [Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning](https://mlsanity.com/paper/1904.10348/) (2019)
- [General 3D Room Layout from a Single View by Render-and-Compare](https://mlsanity.com/paper/2001.02149/) (2020)
- [Joint Embedding of 3D Scan and CAD Objects](https://mlsanity.com/paper/1908.06989/) (2019)

## Built on by (most cited first, 19 of 19)

- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [MonteBoxFinder: Detecting and Filtering Primitives to Fit a Noisy Point Cloud](https://mlsanity.com/paper/2207.14268/) (2022)
- [Automatically Annotating Indoor Images with CAD Models via RGB-D Scans](https://mlsanity.com/paper/2212.11796/) (2022)
- [360-DFPE: Leveraging Monocular 360-Layouts for Direct Floor Plan Estimation](https://mlsanity.com/paper/2112.06180/) (2021)
- [Convex Decomposition of Indoor Scenes](https://mlsanity.com/paper/2307.04246/) (2023)
- [FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos](https://mlsanity.com/paper/2403.15161/) (2024)
- [MCTS with Refinement for Proposals Selection Games in Scene Understanding](https://mlsanity.com/paper/2207.03204/) (2022)
- [Blocks2World: Controlling Realistic Scenes with Editable Primitives](https://mlsanity.com/paper/2307.03847/) (2023)
- [HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scans](https://mlsanity.com/paper/2309.06107/) (2023)
- [Split, Merge, and Refine: Fitting Tight Bounding Boxes via Over-Segmentation and Iterative Search](https://mlsanity.com/paper/2304.04336/) (2023)
- [SSIG: A Visually-Guided Graph Edit Distance for Floor Plan Similarity](https://mlsanity.com/paper/2309.04357/) (2023)
- [Joint stereo 3D object detection and implicit surface reconstruction](https://mlsanity.com/paper/2111.12924/) (2021)
- [Improved Convex Decomposition with Ensembling and Negative Primitives](https://mlsanity.com/paper/2405.19569/) (2024)
- [Inferring Compositional 4D Scenes without Ever Seeing One](https://mlsanity.com/paper/2512.05272/) (2025)
- [FAWN: Floor-And-Walls Normal Regularization for Direct Neural TSDF Reconstruction](https://mlsanity.com/paper/2406.12054/) (2024)
- [PlugTrack: Multi-Perceptive Motion Analysis for Adaptive Fusion in Multi-Object Tracking](https://mlsanity.com/paper/2511.13105/) (2025)
- [PolyLayout: Multi-room Manhattan Layout Estimation](https://mlsanity.com/paper/2608.03323/) (2026)

## Abstract (arXiv)

We explore how a general AI algorithm can be used for 3D scene understanding to reduce the need for training data. More exactly, we propose a modification of the Monte Carlo Tree Search (MCTS) algorithm to retrieve objects and room layouts from noisy RGB-D scans. While MCTS was developed as a game-playing algorithm, we show it can also be used for complex perception problems. Our adapted MCTS algorithm has few easy-to-tune hyperparameters and can optimise general losses. We use it to optimise the posterior probability of objects and room layout hypotheses given the RGB-D data. This results in an analysis-by-synthesis approach that explores the solution space by rendering the current solution and comparing it to the RGB-D observations. To perform this exploration even more efficiently, we propose simple changes to the standard MCTS' tree construction and exploration policy. We demonstrate our approach on the ScanNet dataset. Our method often retrieves configurations that are better than some manual annotations, especially on layouts.