# 3D Sketching using Multi-View Deep Volumetric Prediction

arXiv: https://arxiv.org/abs/1707.08390 · HTML: https://mlsanity.com/paper/1707.08390/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2017.

## Built on by (most cited first, 30 of 37)

- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [SKED: Sketch-guided Text-based 3D Editing](https://mlsanity.com/paper/2303.10735/) (2023)
- [Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches](https://mlsanity.com/paper/2105.06663/) (2021)
- [Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches](https://mlsanity.com/paper/2104.00482/) (2021)
- [Cross-Modal 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2207.11795/) (2022)
- [3D-aware Conditional Image Synthesis](https://mlsanity.com/paper/2302.08509/) (2023)
- [SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling](https://mlsanity.com/paper/2208.06880/) (2022)
- [Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation](https://mlsanity.com/paper/2307.03869/) (2023)
- [DeepCurrents: Learning Implicit Representations of Shapes with Boundaries](https://mlsanity.com/paper/2111.09383/) (2021)
- [3D VR Sketch Guided 3D Shape Prototyping and Exploration](https://mlsanity.com/paper/2306.10830/) (2023)
- [SimpModeling: Sketching Implicit Field to Guide Mesh Modeling for 3D Animalmorphic Head Design](https://mlsanity.com/paper/2108.02548/) (2021)
- [CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration](https://mlsanity.com/paper/2306.08226/) (2023)
- [SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling](https://mlsanity.com/paper/2306.06088/) (2023)
- [Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch](https://mlsanity.com/paper/2201.09367/) (2022)
- [Photo-to-Shape Material Transfer for Diverse Structures](https://mlsanity.com/paper/2205.04018/) (2022)
- [Vitruvio: 3D Building Meshes via Single Perspective Sketches](https://mlsanity.com/paper/2210.13634/) (2022)
- [Sketch2Cloth: Sketch-based 3D Garment Generation with Unsigned Distance Fields](https://mlsanity.com/paper/2303.00167/) (2023)
- [Unsupervised Scene Sketch to Photo Synthesis](https://mlsanity.com/paper/2209.02834/) (2022)
- [SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling](https://mlsanity.com/paper/2307.00804/) (2023)
- [DeepPatent2: A Large-Scale Benchmarking Corpus for Technical Drawing Understanding](https://mlsanity.com/paper/2311.04098/) (2023)
- [GroundUp: Rapid Sketch-Based 3D City Massing](https://mlsanity.com/paper/2407.12739/) (2024)
- [Interactive Sketching of Mannequin Poses](https://mlsanity.com/paper/2212.07098/) (2022)
- [Toon3D: Seeing Cartoons from New Perspectives](https://mlsanity.com/paper/2405.10320/) (2024)
- [VRsketch2Gaussian: 3D VR Sketch Guided 3D Object Generation with Gaussian Splatting](https://mlsanity.com/paper/2503.12383/) (2025)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [Neural UpFlow: A Scene Flow Learning Approach to Increase the Apparent Resolution of Particle-Based Liquids](https://mlsanity.com/paper/2106.05143/) (2021)
- [SSR: Semi-supervised Soft Rasterizer for single-view 2D to 3D Reconstruction](https://mlsanity.com/paper/2108.09593/) (2021)
- [TreeSketchNet: From Sketch To 3D Tree Parameters Generation](https://mlsanity.com/paper/2207.12297/) (2022)
- [DifferSketching: How Differently Do People Sketch 3D Objects?](https://mlsanity.com/paper/2209.08791/) (2022)
- [Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects](https://mlsanity.com/paper/2306.05832/) (2023)

## Abstract (arXiv)

Sketch-based modeling strives to bring the ease and immediacy of drawing to the 3D world. However, while drawings are easy for humans to create, they are very challenging for computers to interpret due to their sparsity and ambiguity. We propose a data-driven approach that tackles this challenge by learning to reconstruct 3D shapes from one or more drawings. At the core of our approach is a deep convolutional neural network (CNN) that predicts occupancy of a voxel grid from a line drawing. This CNN provides us with an initial 3D reconstruction as soon as the user completes a single drawing of the desired shape. We complement this single-view network with an updater CNN that refines an existing prediction given a new drawing of the shape created from a novel viewpoint. A key advantage of our approach is that we can apply the updater iteratively to fuse information from an arbitrary number of viewpoints, without requiring explicit stroke correspondences between the drawings. We train both CNNs by rendering synthetic contour drawings from hand-modeled shape collections as well as from procedurally-generated abstract shapes. Finally, we integrate our CNNs in a minimal modeling interface that allows users to seamlessly draw an object, rotate it to see its 3D reconstruction, and refine it by re-drawing from another vantage point using the 3D reconstruction as guidance. The main strengths of our approach are its robustness to freehand bitmap drawings, its ability to adapt to different object categories, and the continuum it offers between single-view and multi-view sketch-based modeling.