# Robust Shape Fitting for 3D Scene Abstraction

arXiv: https://arxiv.org/abs/2403.10452 · HTML: https://mlsanity.com/paper/2403.10452/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 114 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [ABC: A Big CAD Model Dataset For Geometric Deep Learning](https://mlsanity.com/paper/1812.06216/) (2018)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)

## Built on by (most cited first, 3 of 3)

- [Improved Convex Decomposition with Ensembling and Negative Primitives](https://mlsanity.com/paper/2405.19569/) (2024)
- [Neural Guided Sampling for Quantum Circuit Optimization](https://mlsanity.com/paper/2510.12430/) (2025)
- [Hi-TOPS: Hierarchical Topology-aware Scoring Prior for 3D Part Decomposition](https://mlsanity.com/paper/2608.00767/) (2026)

## Abstract (arXiv)

Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.