# Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches

arXiv: https://arxiv.org/abs/2310.18148 · HTML: https://mlsanity.com/paper/2310.18148/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2023.

## Built on by (most cited first, 7 of 7)

- [Img2CAD: Conditioned 3D CAD Model Generation from Single Image with Structured Visual Geometry](https://mlsanity.com/paper/2410.03417/) (2024)
- [Discrete Latent Perspective Learning for Segmentation and Detection](https://mlsanity.com/paper/2406.10475/) (2024)
- [Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models](https://mlsanity.com/paper/2405.19326/) (2024)
- [Structural and Statistical Texture Knowledge Distillation and Learning for Segmentation](https://mlsanity.com/paper/2503.08043/) (2025)
- [Magic3DSketch: Create Colorful 3D Models From Sketch-Based 3D Modeling Guided by Text and Language-Image Pre-Training](https://mlsanity.com/paper/2407.19225/) (2024)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [Breaking the Box: Enhancing Remote Sensing Image Segmentation with Freehand Sketches](https://mlsanity.com/paper/2503.12191/) (2025)

## Abstract (arXiv)

The emerging trend of AR/VR places great demands on 3D content. However, most existing software requires expertise and is difficult for novice users to use. In this paper, we aim to create sketch-based modeling tools for user-friendly 3D modeling. We introduce Reality3DSketch with a novel application of an immersive 3D modeling experience, in which a user can capture the surrounding scene using a monocular RGB camera and can draw a single sketch of an object in the real-time reconstructed 3D scene. A 3D object is generated and placed in the desired location, enabled by our novel neural network with the input of a single sketch. Our neural network can predict the pose of a drawing and can turn a single sketch into a 3D model with view and structural awareness, which addresses the challenge of sparse sketch input and view ambiguity. We conducted extensive experiments synthetic and real-world datasets and achieved state-of-the-art (SOTA) results in both sketch view estimation and 3D modeling performance. According to our user study, our method of performing 3D modeling in a scene is >5x faster than conventional methods. Users are also more satisfied with the generated 3D model than the results of existing methods.