# Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding

arXiv: https://arxiv.org/abs/2504.13580 · HTML: https://mlsanity.com/paper/2504.13580/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Builds on (most cited first, 30 of 35)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels](https://mlsanity.com/paper/2312.17232/) (2023)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [Scan2Mesh: From Unstructured Range Scans to 3D Meshes](https://mlsanity.com/paper/1811.10464/) (2018)
- [Learning 3D Shape Completion under Weak Supervision](https://mlsanity.com/paper/1805.07290/) (2018)
- [End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans](https://mlsanity.com/paper/1906.04201/) (2019)
- [A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images](https://mlsanity.com/paper/1903.04704/) (2019)
- [Point-Cloud Completion with Pretrained Text-to-image Diffusion Models](https://mlsanity.com/paper/2306.10533/) (2023)
- [SkeletonNet: A Topology-Preserving Solution for Learning Mesh Reconstruction of Object Surfaces from RGB Images](https://mlsanity.com/paper/2008.05742/) (2020)
- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories](https://mlsanity.com/paper/2206.04916/) (2022)
- [CAD-Estate: Large-scale CAD Model Annotation in RGB Videos](https://mlsanity.com/paper/2306.09011/) (2023)
- [LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories](https://mlsanity.com/paper/2311.12174/) (2023)
- [Automatically Annotating Indoor Images with CAD Models via RGB-D Scans](https://mlsanity.com/paper/2212.11796/) (2022)
- [FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos](https://mlsanity.com/paper/2403.15161/) (2024)

## Built on by (most cited first, 2 of 2)

- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment](https://mlsanity.com/paper/2607.15058/) (2026)

## Abstract (arXiv)

High-level 3D scene understanding is essential in many applications. However, the challenges of generating accurate 3D annotations make development of deep learning models difficult. We turn to recent advancements in automatic retrieval of synthetic CAD models, and show that data generated by such methods can be used as high-quality ground truth for training supervised deep learning models. More exactly, we employ a pipeline akin to the one previously used to automatically annotate objects in ScanNet scenes with their 9D poses and CAD models. This time, we apply it to the recent ScanNet++ v1 dataset, which previously lacked such annotations. Our findings demonstrate that it is not only possible to train deep learning models on these automatically-obtained annotations but that the resulting models outperform those trained on manually annotated data. We validate this on two distinct tasks: point cloud completion and single-view CAD model retrieval and alignment. Our results underscore the potential of automatic 3D annotations to enhance model performance while significantly reducing annotation costs. To support future research in 3D scene understanding, we will release our annotations, which we call SCANnotate++, along with our trained models.