# MCTS with Refinement for Proposals Selection Games in Scene Understanding

arXiv: https://arxiv.org/abs/2207.03204 · HTML: https://mlsanity.com/paper/2207.03204/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image](https://mlsanity.com/paper/1812.04072/) (2018)
- [PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1804.06278/) (2018)
- [Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense](https://mlsanity.com/paper/1909.01507/) (2019)
- [Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding](https://mlsanity.com/paper/1902.09777/) (2019)
- [Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation](https://mlsanity.com/paper/1810.13049/) (2018)
- [SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans](https://mlsanity.com/paper/2003.12622/) (2020)
- [RoomNet: End-to-End Room Layout Estimation](https://mlsanity.com/paper/1703.06241/) (2017)
- [Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path](https://mlsanity.com/paper/1908.06702/) (2019)
- [FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans](https://mlsanity.com/paper/1804.00090/) (2018)
- [PlaneTR: Structure-Guided Transformers for 3D Plane Recovery](https://mlsanity.com/paper/2107.13108/) (2021)
- [HEAT: Holistic Edge Attention Transformer for Structured Reconstruction](https://mlsanity.com/paper/2111.15143/) (2021)
- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [General 3D Room Layout from a Single View by Render-and-Compare](https://mlsanity.com/paper/2001.02149/) (2020)
- [Monte Carlo Scene Search for 3D Scene Understanding](https://mlsanity.com/paper/2103.07969/) (2021)
- [GeoLayout: Geometry Driven Room Layout Estimation Based on Depth Maps of Planes](https://mlsanity.com/paper/2008.06286/) (2020)
- [Thinking Outside the Box: Generation of Unconstrained 3D Room Layouts](https://mlsanity.com/paper/1905.03105/) (2019)
- [Smart Hypothesis Generation for Efficient and Robust Room Layout Estimation](https://mlsanity.com/paper/1910.12257/) (2019)
- [Edge-Semantic Learning Strategy for Layout Estimation in Indoor Environment](https://mlsanity.com/paper/1901.00621/) (2019)

## Built on by (most cited first, 4 of 4)

- [PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape Reconstruction](https://mlsanity.com/paper/2404.10620/) (2024)
- [HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scans](https://mlsanity.com/paper/2309.06107/) (2023)
- [Split, Merge, and Refine: Fitting Tight Bounding Boxes via Over-Segmentation and Iterative Search](https://mlsanity.com/paper/2304.04336/) (2023)
- [Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding](https://mlsanity.com/paper/2504.13580/) (2025)

## Abstract (arXiv)

We propose a novel method applicable in many scene understanding problems that adapts the Monte Carlo Tree Search (MCTS) algorithm, originally designed to learn to play games of high-state complexity. From a generated pool of proposals, our method jointly selects and optimizes proposals that minimize the objective term. In our first application for floor plan reconstruction from point clouds, our method selects and refines the room proposals, modelled as 2D polygons, by optimizing on an objective function combining the fitness as predicted by a deep network and regularizing terms on the room shapes. We also introduce a novel differentiable method for rendering the polygonal shapes of these proposals. Our evaluations on the recent and challenging Structured3D and Floor-SP datasets show significant improvements over the state-of-the-art, without imposing hard constraints nor assumptions on the floor plan configurations. In our second application, we extend our approach to reconstruct general 3D room layouts from a color image and obtain accurate room layouts. We also show that our differentiable renderer can easily be extended for rendering 3D planar polygons and polygon embeddings. Our method shows high performance on the Matterport3D-Layout dataset, without introducing hard constraints on room layout configurations.