# Zero in on Shape: A Generic 2D-3D Instance Similarity Metric learned from Synthetic Data

arXiv: https://arxiv.org/abs/2108.04091 · HTML: https://mlsanity.com/paper/2108.04091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [MeshNet: Mesh Neural Network for 3D Shape Representation](https://mlsanity.com/paper/1811.11424/) (2018)
- [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild](https://mlsanity.com/paper/1803.11493/) (2018)
- [LCD: Learned Cross-Domain Descriptors for 2D-3D Matching](https://mlsanity.com/paper/1911.09326/) (2019)
- [Location Field Descriptors: Single Image 3D Model Retrieval in the Wild](https://mlsanity.com/paper/1908.02853/) (2019)
- [Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views](https://mlsanity.com/paper/1512.02497/) (2015)
- [Focused LRP: Explainable AI for Face Morphing Attack Detection](https://mlsanity.com/paper/2103.14697/) (2021)

## Built on by (most cited first, 1 of 1)

- [SPVLoc: Semantic Panoramic Viewport Matching for 6D Camera Localization in Unseen Environments](https://mlsanity.com/paper/2404.10527/) (2024)

## Abstract (arXiv)

We present a network architecture which compares RGB images and untextured 3D models by the similarity of the represented shape. Our system is optimised for zero-shot retrieval, meaning it can recognise shapes never shown in training. We use a view-based shape descriptor and a siamese network to learn object geometry from pairs of 3D models and 2D images. Due to scarcity of datasets with exact photograph-mesh correspondences, we train our network with only synthetic data. Our experiments investigate the effect of different qualities and quantities of training data on retrieval accuracy and present insights from bridging the domain gap. We show that increasing the variety of synthetic data improves retrieval accuracy and that our system's performance in zero-shot mode can match that of the instance-aware mode, as far as narrowing down the search to the top 10% of objects.