# Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images

arXiv: https://arxiv.org/abs/2105.02047 · HTML: https://mlsanity.com/paper/2105.02047/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 70 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [Graph-Cut RANSAC](https://mlsanity.com/paper/1706.00984/) (2017)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)

## Built on by (most cited first, 17 of 17)

- [Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives](https://mlsanity.com/paper/2307.05473/) (2023)
- [Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning](https://mlsanity.com/paper/2405.08054/) (2024)
- [NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping](https://mlsanity.com/paper/2111.15207/) (2021)
- [Iterative Superquadric Recomposition of 3D Objects from Multiple Views](https://mlsanity.com/paper/2309.02102/) (2023)
- [AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings](https://mlsanity.com/paper/2406.08960/) (2024)
- [MonteBoxFinder: Detecting and Filtering Primitives to Fit a Noisy Point Cloud](https://mlsanity.com/paper/2207.14268/) (2022)
- [PartSDF: Part-Based Implicit Neural Representation for Composite 3D Shape Parametrization and Optimization](https://mlsanity.com/paper/2502.12985/) (2025)
- [Convex Decomposition of Indoor Scenes](https://mlsanity.com/paper/2307.04246/) (2023)
- [Primitive-based 3D Human-Object Interaction Modelling and Programming](https://mlsanity.com/paper/2312.10714/) (2023)
- [PARSAC: Accelerating Robust Multi-Model Fitting with Parallel Sample Consensus](https://mlsanity.com/paper/2401.14919/) (2024)
- [HybridSDF: Combining Deep Implicit Shapes and Geometric Primitives for 3D Shape Representation and Manipulation](https://mlsanity.com/paper/2109.10767/) (2021)
- [Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scans](https://mlsanity.com/paper/2304.09704/) (2023)
- [Blocks2World: Controlling Realistic Scenes with Editable Primitives](https://mlsanity.com/paper/2307.03847/) (2023)
- [Robust Shape Fitting for 3D Scene Abstraction](https://mlsanity.com/paper/2403.10452/) (2024)
- [Generative Blocks World: Moving Things Around in Pictures](https://mlsanity.com/paper/2506.20703/) (2025)
- [Improved Convex Decomposition with Ensembling and Negative Primitives](https://mlsanity.com/paper/2405.19569/) (2024)
- [A 3D Generation Framework from Cross Modality to Parameterized Primitive](https://mlsanity.com/paper/2510.08656/) (2025)

## Abstract (arXiv)

Humans perceive and construct the surrounding world as an arrangement of simple parametric models. In particular, man-made environments commonly consist of volumetric primitives such as cuboids or cylinders. Inferring these primitives is an important step to attain high-level, abstract scene descriptions. Previous approaches directly estimate shape parameters from a 2D or 3D input, and are only able to reproduce simple objects, yet unable to accurately parse more complex 3D scenes. In contrast, we propose a robust estimator for primitive fitting, which can meaningfully abstract real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to 3D features, such as a depth map. We condition the network on previously detected parts of the scene, thus parsing it one-by-one. To obtain 3D features from a single RGB image, we additionally optimise a feature extraction CNN in an end-to-end manner. However, naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene behind. We thus propose an occlusion-aware distance metric correctly handling opaque scenes. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the challenging NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.