# FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos

arXiv: https://arxiv.org/abs/2403.15161 · HTML: https://mlsanity.com/paper/2403.15161/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://mlsanity.com/paper/2006.05682/) (2020)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection](https://mlsanity.com/paper/2112.00322/) (2021)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [Generative Sparse Detection Networks for 3D Single-shot Object Detection](https://mlsanity.com/paper/2006.12356/) (2020)
- [Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2104.06114/) (2021)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans](https://mlsanity.com/paper/2003.12622/) (2020)
- [End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans](https://mlsanity.com/paper/1906.04201/) (2019)
- [RBGNet: Ray-based Grouping for 3D Object Detection](https://mlsanity.com/paper/2204.02251/) (2022)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos](https://mlsanity.com/paper/2012.04641/) (2020)
- [TR3D: Towards Real-Time Indoor 3D Object Detection](https://mlsanity.com/paper/2302.02858/) (2023)
- [ODAM: Object Detection, Association, and Mapping using Posed RGB Video](https://mlsanity.com/paper/2108.10165/) (2021)
- [Joint Embedding of 3D Scan and CAD Objects](https://mlsanity.com/paper/1908.06989/) (2019)
- [Monte Carlo Scene Search for 3D Scene Understanding](https://mlsanity.com/paper/2103.07969/) (2021)
- [RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers](https://mlsanity.com/paper/2203.13296/) (2022)
- [Automatically Annotating Indoor Images with CAD Models via RGB-D Scans](https://mlsanity.com/paper/2212.11796/) (2022)

## Built on by (most cited first, 7 of 7)

- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling](https://mlsanity.com/paper/2411.19492/) (2024)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation](https://mlsanity.com/paper/2602.22785/) (2026)
- [Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding](https://mlsanity.com/paper/2504.13580/) (2025)
- [BRepCLIP: Contrastive Multimodal Pretraining on BRep Primitives for CAD Understanding](https://mlsanity.com/paper/2606.05515/) (2026)
- [CAOA -- Completion-Assisted Object-CAD Alignment](https://mlsanity.com/paper/2606.18429/) (2026)

## Abstract (arXiv)

Digitising the 3D world into a clean, CAD model-based representation has important applications for augmented reality and robotics. Current state-of-the-art methods are computationally intensive as they individually encode each detected object and optimise CAD alignments in a second stage. In this work, we propose FastCAD, a real-time method that simultaneously retrieves and aligns CAD models for all objects in a given scene. In contrast to previous works, we directly predict alignment parameters and shape embeddings. We achieve high-quality shape retrievals by learning CAD embeddings in a contrastive learning framework and distilling those into FastCAD. Our single-stage method accelerates the inference time by a factor of 50 compared to other methods operating on RGB-D scans while outperforming them on the challenging Scan2CAD alignment benchmark. Further, our approach collaborates seamlessly with online 3D reconstruction techniques. This enables the real-time generation of precise CAD model-based reconstructions from videos at 10 FPS. Doing so, we significantly improve the Scan2CAD alignment accuracy in the video setting from 43.0% to 48.2% and the reconstruction accuracy from 22.9% to 29.6%.