# End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans

arXiv: https://arxiv.org/abs/1906.04201 · HTML: https://mlsanity.com/paper/1906.04201/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 30 of 40)

- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image](https://mlsanity.com/paper/2311.18610/) (2023)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [FG2: Fine-Grained Cross-View Localization by Fine-Grained Feature Matching](https://mlsanity.com/paper/2503.18725/) (2025)
- [Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences](https://mlsanity.com/paper/2404.06337/) (2024)
- [OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects](https://mlsanity.com/paper/2407.08711/) (2024)
- [ShapeR: Robust Conditional 3D Shape Generation from Casual Captures](https://mlsanity.com/paper/2601.11514/) (2026)
- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [Monte Carlo Scene Search for 3D Scene Understanding](https://mlsanity.com/paper/2103.07969/) (2021)
- [Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement](https://mlsanity.com/paper/2203.05238/) (2022)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairs](https://mlsanity.com/paper/2207.00826/) (2022)
- [Optimal Pose and Shape Estimation for Category-level 3D Object Perception](https://mlsanity.com/paper/2104.08383/) (2021)
- [ObjectMatch: Robust Registration using Canonical Object Correspondences](https://mlsanity.com/paper/2212.01985/) (2022)
- [U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds](https://mlsanity.com/paper/2308.06383/) (2023)
- [ELLIPSDF: Joint Object Pose and Shape Optimization with a Bi-level Ellipsoid and Signed Distance Function Description](https://mlsanity.com/paper/2108.00355/) (2021)
- [Automatically Annotating Indoor Images with CAD Models via RGB-D Scans](https://mlsanity.com/paper/2212.11796/) (2022)
- [SCARP: 3D Shape Completion in ARbitrary Poses for Improved Grasping](https://mlsanity.com/paper/2301.07213/) (2023)
- [ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation](https://mlsanity.com/paper/2311.11106/) (2023)
- [Multi-instance Point Cloud Registration by Efficient Correspondence Clustering](https://mlsanity.com/paper/2111.14582/) (2021)
- [Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints](https://mlsanity.com/paper/2206.12498/) (2022)
- [FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos](https://mlsanity.com/paper/2403.15161/) (2024)
- [Towards 3D Scene Understanding by Referring Synthetic Models](https://mlsanity.com/paper/2203.10546/) (2022)
- [KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation](https://mlsanity.com/paper/2403.10099/) (2024)
- [Towards 3D VR-Sketch to 3D Shape Retrieval](https://mlsanity.com/paper/2209.10020/) (2022)
- [Accurate Instance-Level CAD Model Retrieval in a Large-Scale Database](https://mlsanity.com/paper/2207.01339/) (2022)
- [HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scans](https://mlsanity.com/paper/2309.06107/) (2023)
- [Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training](https://mlsanity.com/paper/2309.16956/) (2023)
- [Weakly-Supervised End-to-End CAD Retrieval to Scan Objects](https://mlsanity.com/paper/2203.12873/) (2022)

## Abstract (arXiv)

We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry. Our main contribution lies in formulating a differentiable Procrustes alignment that is paired with a symmetry-aware dense object correspondence prediction. To simultaneously align CAD models to all the objects of a scanned scene, our approach detects object locations, then predicts symmetry-aware dense object correspondences between scan and CAD geometry in a unified object space, as well as a nearest neighbor CAD model, both of which are then used to inform a differentiable Procrustes alignment. Our approach operates in a fully-convolutional fashion, enabling alignment of CAD models to the objects of a scan in a single forward pass. This enables our method to outperform state-of-the-art approaches by 19.04% for CAD model alignment to scans, with ≈ 250× faster runtime than previous data-driven approaches.