mlsanityData to 7 Oct 2026

Monte Carlo Scene Search for 3D Scene Understanding

arXiv 2103.07969 ↗First posted 14 Mar 2021cs.CV cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We explore how a general AI algorithm can be used for 3D scene understanding to reduce the need for training data. More exactly, we propose a modification of the Monte Carlo Tree Search (MCTS) algorithm to retrieve objects and room layouts from noisy RGB-D scans. While MCTS was developed as a game-playing algorithm, we show it can also be used for complex perception problems. Our adapted MCTS algorithm has few easy-to-tune hyperparameters and can optimise general losses. We use it to optimise the posterior probability of objects and room layout hypotheses given the RGB-D data. This results in an analysis-by-synthesis approach that explores the solution space by rendering the current solution and comparing it to the RGB-D observations. To perform this exploration even more efficiently, we propose simple changes to the standard MCTS' tree construction and exploration policy. We demonstrate our approach on the ScanNet dataset. Our method often retrieves configurations that are better than some manual annotations, especially on layouts.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 32 of its 63 references that are in the map. First posted on arXiv 14 Mar 2021.

Builds on

32 of 63 references.

References · 32

  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  6. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  7. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  8. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  9. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  10. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  11. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  12. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
20 more
  1. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  2. 2019HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentationcited by 107
  3. 2018LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Imagecited by 94
  4. 2016IM2CADcited by 78
  5. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  6. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  7. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  8. 2018Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimationcited by 51
  9. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  10. 2020SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scanscited by 44
  11. 2019End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scanscited by 40
  12. 2017RoomNet: End-to-End Room Layout Estimationcited by 38
  13. 2018FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scanscited by 35
  14. 20183D Pose Estimation and 3D Model Retrieval for Objects in the Wildcited by 31
  15. 20193D-RelNet: Joint Object and Relational Network for 3D Predictioncited by 28
  16. 2019Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planningcited by 23
  17. 2020General 3D Room Layout from a Single View by Render-and-Comparecited by 21
  18. 2019Joint Embedding of 3D Scan and CAD Objectscited by 19
  19. 2020GeoLayout: Geometry Driven Room Layout Estimation Based on Depth Maps of Planescited by 12
  20. 2019Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methodscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 5; 2024: 3; 2025: 3; 2026: 1.

19 papers.

Sort:

  1. 2021MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Planscited by 26
  2. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  3. 2022Point Scene Understanding via Disentangled Instance Mesh Reconstructioncited by 20
  4. 2022MonteBoxFinder: Detecting and Filtering Primitives to Fit a Noisy Point Cloudcited by 11
  5. 2022Automatically Annotating Indoor Images with CAD Models via RGB-D Scanscited by 9
  6. 2021360-DFPE: Leveraging Monocular 360-Layouts for Direct Floor Plan Estimationcited by 8
  7. 2024FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videoscited by 7
  8. 2023Convex Decomposition of Indoor Scenescited by 6
  9. 2022MCTS with Refinement for Proposals Selection Games in Scene Understandingcited by 4
  10. 2023Blocks2World: Controlling Realistic Scenes with Editable Primitivescited by 4
  11. 2023HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scanscited by 3
  12. 2023Split, Merge, and Refine: Fitting Tight Bounding Boxes via Over-Segmentation and Iterative Searchcited by 2
7 more
  1. 2023SSIG: A Visually-Guided Graph Edit Distance for Floor Plan Similaritycited by 2
  2. 2021Joint stereo 3D object detection and implicit surface reconstructioncited by 1
  3. 2025Inferring Compositional 4D Scenes without Ever Seeing Onecited by 1
  4. 2024Improved Convex Decomposition with Ensembling and Negative Primitivescited by 0
  5. 2024FAWN: Floor-And-Walls Normal Regularization for Direct Neural TSDF Reconstructioncited by 0
  6. 2025PlugTrack: Multi-Perceptive Motion Analysis for Adaptive Fusion in Multi-Object Trackingcited by 0
  7. 2026PolyLayout: Multi-room Manhattan Layout Estimationcited by 0