# HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scans

arXiv: https://arxiv.org/abs/2309.06107 · HTML: https://mlsanity.com/paper/2309.06107/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Learning Gradient Fields for Shape Generation](https://mlsanity.com/paper/2008.06520/) (2020)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans](https://mlsanity.com/paper/1906.04201/) (2019)
- [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild](https://mlsanity.com/paper/1803.11493/) (2018)
- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [Joint Embedding of 3D Scan and CAD Objects](https://mlsanity.com/paper/1908.06989/) (2019)
- [Monte Carlo Scene Search for 3D Scene Understanding](https://mlsanity.com/paper/2103.07969/) (2021)
- [Location Field Descriptors: Single Image 3D Model Retrieval in the Wild](https://mlsanity.com/paper/1908.02853/) (2019)
- [Understanding deep features with computer-generated imagery](https://mlsanity.com/paper/1506.01151/) (2015)
- [Automatically Annotating Indoor Images with CAD Models via RGB-D Scans](https://mlsanity.com/paper/2212.11796/) (2022)
- [Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views](https://mlsanity.com/paper/1512.02497/) (2015)
- [A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition](https://mlsanity.com/paper/1504.02764/) (2015)
- [MCTS with Refinement for Proposals Selection Games in Scene Understanding](https://mlsanity.com/paper/2207.03204/) (2022)
- [Accurate Instance-Level CAD Model Retrieval in a Large-Scale Database](https://mlsanity.com/paper/2207.01339/) (2022)

## Built on by (most cited first, 3 of 3)

- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape Reconstruction](https://mlsanity.com/paper/2404.10620/) (2024)
- [Leveraging Automatic CAD Annotations for Supervised Learning in 3D Scene Understanding](https://mlsanity.com/paper/2504.13580/) (2025)

## Abstract (arXiv)

We present an automated and efficient approach for retrieving high-quality CAD models of objects and their poses in a scene captured by a moving RGB-D camera. We first investigate various objective functions to measure similarity between a candidate CAD object model and the available data, and the best objective function appears to be a "render-and-compare" method comparing depth and mask rendering. We thus introduce a fast-search method that approximates an exhaustive search based on this objective function for simultaneously retrieving the object category, a CAD model, and the pose of an object given an approximate 3D bounding box. This method involves a search tree that organizes the CAD models and object properties including object category and pose for fast retrieval and an algorithm inspired by Monte Carlo Tree Search, that efficiently searches this tree. We show that this method retrieves CAD models that fit the real objects very well, with a speed-up factor of 10x to 120x compared to exhaustive search.