# Structure-Aware 3D VR Sketch to 3D Shape Retrieval

arXiv: https://arxiv.org/abs/2209.09043 · HTML: https://mlsanity.com/paper/2209.09043/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [Neural Cages for Detail-Preserving 3D Deformations](https://mlsanity.com/paper/1912.06395/) (2019)
- [Triplet-Center Loss for Multi-View 3D Object Retrieval](https://mlsanity.com/paper/1803.06189/) (2018)
- [Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval](https://mlsanity.com/paper/1703.05605/) (2017)
- [KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control](https://mlsanity.com/paper/2104.11224/) (2021)
- [Joint Learning of 3D Shape Retrieval and Deformation](https://mlsanity.com/paper/2101.07889/) (2021)
- [Sketch-based 3D Shape Retrieval using Convolutional Neural Networks](https://mlsanity.com/paper/1504.03504/) (2015)
- [Unsupervised Learning of Intrinsic Structural Representation Points](https://mlsanity.com/paper/2003.01661/) (2020)
- [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild](https://mlsanity.com/paper/1803.11493/) (2018)
- [Deformation-Aware 3D Model Embedding and Retrieval](https://mlsanity.com/paper/2004.01228/) (2020)
- [Learning Generative Models of Shape Handles](https://mlsanity.com/paper/2004.03028/) (2020)
- [3D Object Instance Recognition and Pose Estimation Using Triplet Loss with Dynamic Margin](https://mlsanity.com/paper/1904.04854/) (2019)
- [A weakly supervised adaptive triplet loss for deep metric learning](https://mlsanity.com/paper/1909.12939/) (2019)
- [Deep Ranking with Adaptive Margin Triplet Loss](https://mlsanity.com/paper/2107.06187/) (2021)
- [Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality](https://mlsanity.com/paper/2105.02139/) (2021)

## Built on by (most cited first, 7 of 7)

- [It's All About Your Sketch: Democratising Sketch Control in Diffusion Models](https://mlsanity.com/paper/2403.07234/) (2024)
- [3D VR Sketch Guided 3D Shape Prototyping and Exploration](https://mlsanity.com/paper/2306.10830/) (2023)
- [SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling](https://mlsanity.com/paper/2307.00804/) (2023)
- [SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models](https://mlsanity.com/paper/2503.14129/) (2025)
- [Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval](https://mlsanity.com/paper/2306.08541/) (2023)
- [MS2Mesh-XR: Multi-modal Sketch-to-Mesh Generation in XR Environments](https://mlsanity.com/paper/2412.09008/) (2024)
- [SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches](https://mlsanity.com/paper/2501.16022/) (2025)

## Abstract (arXiv)

We study the practical task of fine-grained 3D-VR-sketch-based 3D shape retrieval. This task is of particular interest as 2D sketches were shown to be effective queries for 2D images. However, due to the domain gap, it remains hard to achieve strong performance in 3D shape retrieval from 2D sketches. Recent work demonstrated the advantage of 3D VR sketching on this task. In our work, we focus on the challenge caused by inherent inaccuracies in 3D VR sketches. We observe that retrieval results obtained with a triplet loss with a fixed margin value, commonly used for retrieval tasks, contain many irrelevant shapes and often just one or few with a similar structure to the query. To mitigate this problem, we for the first time draw a connection between adaptive margin values and shape similarities. In particular, we propose to use a triplet loss with an adaptive margin value driven by a "fitting gap", which is the similarity of two shapes under structure-preserving deformations. We also conduct a user study which confirms that this fitting gap is indeed a suitable criterion to evaluate the structural similarity of shapes. Furthermore, we introduce a dataset of 202 VR sketches for 202 3D shapes drawn from memory rather than from observation. The code and data are available at https://github.com/Rowl1ng/Structure-Aware-VR-Sketch-Shape-Retrieval.