# ROCA: Robust CAD Model Retrieval and Alignment from a Single Image

arXiv: https://arxiv.org/abs/2112.01988 · HTML: https://mlsanity.com/paper/2112.01988/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 54 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1808.02201/) (2018)

## Built on by (most cited first, 30 of 67)

- [MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation](https://mlsanity.com/paper/2412.03558/) (2024)
- [CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image](https://mlsanity.com/paper/2502.12894/) (2025)
- [PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers](https://mlsanity.com/paper/2506.05573/) (2025)
- [SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass](https://mlsanity.com/paper/2508.15769/) (2025)
- [DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image](https://mlsanity.com/paper/2311.18610/) (2023)
- [ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model](https://mlsanity.com/paper/2311.17618/) (2023)
- [GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors](https://mlsanity.com/paper/2409.16294/) (2024)
- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera](https://mlsanity.com/paper/2408.03225/) (2024)
- [CADDreamer: CAD Object Generation from Single-view Images](https://mlsanity.com/paper/2502.20732/) (2025)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusion](https://mlsanity.com/paper/2507.22825/) (2025)
- [OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects](https://mlsanity.com/paper/2407.08711/) (2024)
- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [Visual Localization using Imperfect 3D Models from the Internet](https://mlsanity.com/paper/2304.05947/) (2023)
- [SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB image](https://mlsanity.com/paper/2210.01044/) (2022)
- [WorldGen: From Text to Traversable and Interactive 3D Worlds](https://mlsanity.com/paper/2511.16825/) (2025)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [ObjectMatch: Robust Registration using Canonical Object Correspondences](https://mlsanity.com/paper/2212.01985/) (2022)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds](https://mlsanity.com/paper/2308.06383/) (2023)
- [Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling](https://mlsanity.com/paper/2411.19492/) (2024)
- [3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image](https://mlsanity.com/paper/2604.04406/) (2026)
- [SDFit: 3D Object Pose and Shape by Fitting a Morphable SDF to a Single Image](https://mlsanity.com/paper/2409.16178/) (2024)
- [ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation](https://mlsanity.com/paper/2311.11106/) (2023)
- [Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects](https://mlsanity.com/paper/2401.00405/) (2023)
- [Coherent 3D Scene Diffusion From a Single RGB Image](https://mlsanity.com/paper/2412.10294/) (2024)
- [Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes](https://mlsanity.com/paper/2404.04557/) (2024)
- [Reconstructing Animals and the Wild](https://mlsanity.com/paper/2411.18807/) (2024)
- [UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation](https://mlsanity.com/paper/2411.19292/) (2024)

## Abstract (arXiv)

We present ROCA, a novel end-to-end approach that retrieves and aligns 3D CAD models from a shape database to a single input image. This enables 3D perception of an observed scene from a 2D RGB observation, characterized as a lightweight, compact, clean CAD representation. Core to our approach is our differentiable alignment optimization based on dense 2D-3D object correspondences and Procrustes alignment. ROCA can thus provide a robust CAD alignment while simultaneously informing CAD retrieval by leveraging the 2D-3D correspondences to learn geometrically similar CAD models. Experiments on challenging, real-world imagery from ScanNet show that ROCA significantly improves on state of the art, from 9.5% to 17.6% in retrieval-aware CAD alignment accuracy.