# Leveraging Geometry for Shape Estimation from a Single RGB Image

arXiv: https://arxiv.org/abs/2111.05615 · HTML: https://mlsanity.com/paper/2111.05615/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks](https://mlsanity.com/paper/1909.00321/) (2019)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos](https://mlsanity.com/paper/2012.04641/) (2020)
- [Location Field Descriptors: Single Image 3D Model Retrieval in the Wild](https://mlsanity.com/paper/1908.02853/) (2019)
- [Inferring Point Clouds from Single Monocular Images by Depth Intermediation](https://mlsanity.com/paper/1812.01402/) (2018)

## Built on by (most cited first, 4 of 4)

- [SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB image](https://mlsanity.com/paper/2210.01044/) (2022)
- [FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos](https://mlsanity.com/paper/2403.15161/) (2024)
- [Sparse Multi-Object Render-and-Compare](https://mlsanity.com/paper/2310.11184/) (2023)
- [Glissando-Net: Deep sinGLe vIew category level poSe eStimation ANd 3D recOnstruction](https://mlsanity.com/paper/2501.14896/) (2025)

## Abstract (arXiv)

Predicting 3D shapes and poses of static objects from a single RGB image is an important research area in modern computer vision. Its applications range from augmented reality to robotics and digital content creation. Typically this task is performed through direct object shape and pose predictions which is inaccurate. A promising research direction ensures meaningful shape predictions by retrieving CAD models from large scale databases and aligning them to the objects observed in the image. However, existing work does not take the object geometry into account, leading to inaccurate object pose predictions, especially for unseen objects. In this work we demonstrate how cross-domain keypoint matches from an RGB image to a rendered CAD model allow for more precise object pose predictions compared to ones obtained through direct predictions. We further show that keypoint matches can not only be used to estimate the pose of an object, but also to modify the shape of the object itself. This is important as the accuracy that can be achieved with object retrieval alone is inherently limited to the available CAD models. Allowing shape adaptation bridges the gap between the retrieved CAD model and the observed shape. We demonstrate our approach on the challenging Pix3D dataset. The proposed geometric shape prediction improves the AP mesh over the state-of-the-art from 33.2 to 37.8 on seen objects and from 8.2 to 17.1 on unseen objects. Furthermore, we demonstrate more accurate shape predictions without closely matching CAD models when following the proposed shape adaptation. Code is publicly available at https://github.com/florianlanger/leveraging_geometry_for_shape_estimation .