# Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches

arXiv: https://arxiv.org/abs/2104.00482 · HTML: https://mlsanity.com/paper/2104.00482/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [MeshSDF: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2006.03997/) (2020)
- [FroDO: From Detections to 3D Objects](https://mlsanity.com/paper/2005.05125/) (2020)
- [3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks](https://mlsanity.com/paper/1707.06375/) (2017)
- [3D Sketching using Multi-View Deep Volumetric Prediction](https://mlsanity.com/paper/1707.08390/) (2017)
- [Coupling Explicit and Implicit Surface Representations for Generative 3D Modeling](https://mlsanity.com/paper/2007.10294/) (2020)
- [Deep Sketch-Based Modeling: Tips and Tricks](https://mlsanity.com/paper/2011.06133/) (2020)
- [UCLID-Net: Single View Reconstruction in Object Space](https://mlsanity.com/paper/2006.03817/) (2020)

## Built on by (most cited first, 30 of 51)

- [Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation](https://mlsanity.com/paper/2501.12202/) (2025)
- [Locally Attentional SDF Diffusion for Controllable 3D Shape Generation](https://mlsanity.com/paper/2305.04461/) (2023)
- [Control3D: Towards Controllable Text-to-3D Generation](https://mlsanity.com/paper/2311.05461/) (2023)
- [Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation](https://mlsanity.com/paper/2308.02874/) (2023)
- [Text and Image Guided 3D Avatar Generation and Manipulation](https://mlsanity.com/paper/2202.06079/) (2022)
- [NANO3D: A Training-Free Approach for Efficient 3D Editing Without Masks](https://mlsanity.com/paper/2510.15019/) (2025)
- [Cross-Modal 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2207.11795/) (2022)
- [Make-A-Shape: a Ten-Million-scale 3D Shape Model](https://mlsanity.com/paper/2401.11067/) (2024)
- [3Doodle: Compact Abstraction of Objects with 3D Strokes](https://mlsanity.com/paper/2402.03690/) (2024)
- [Local 3D Editing via 3D Distillation of CLIP Knowledge](https://mlsanity.com/paper/2306.12570/) (2023)
- [DeepMesh: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2106.11795/) (2021)
- [SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling](https://mlsanity.com/paper/2208.06880/) (2022)
- [Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation](https://mlsanity.com/paper/2307.03869/) (2023)
- [Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes](https://mlsanity.com/paper/2312.04043/) (2023)
- [3D VR Sketch Guided 3D Shape Prototyping and Exploration](https://mlsanity.com/paper/2306.10830/) (2023)
- [Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings](https://mlsanity.com/paper/2411.08017/) (2024)
- [CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration](https://mlsanity.com/paper/2306.08226/) (2023)
- [SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling](https://mlsanity.com/paper/2306.06088/) (2023)
- [CNS-Edit: 3D Shape Editing via Coupled Neural Shape Optimization](https://mlsanity.com/paper/2402.02313/) (2024)
- [HybridSDF: Combining Deep Implicit Shapes and Geometric Primitives for 3D Shape Representation and Manipulation](https://mlsanity.com/paper/2109.10767/) (2021)
- [Vitruvio: 3D Building Meshes via Single Perspective Sketches](https://mlsanity.com/paper/2210.13634/) (2022)
- [Sketch2Cloth: Sketch-based 3D Garment Generation with Unsigned Distance Fields](https://mlsanity.com/paper/2303.00167/) (2023)
- [Deep3DSketch+: Rapid 3D Modeling from Single Free-hand Sketches](https://mlsanity.com/paper/2309.13006/) (2023)
- [Magic3DSketch: Create Colorful 3D Models From Sketch-Based 3D Modeling Guided by Text and Language-Image Pre-Training](https://mlsanity.com/paper/2407.19225/) (2024)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [Neural Implicit Shape Editing using Boundary Sensitivity](https://mlsanity.com/paper/2304.12951/) (2023)
- [SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling](https://mlsanity.com/paper/2307.00804/) (2023)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [GroundUp: Rapid Sketch-Based 3D City Massing](https://mlsanity.com/paper/2407.12739/) (2024)
- [Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval](https://mlsanity.com/paper/2306.08541/) (2023)

## Abstract (arXiv)

Reconstructing 3D shape from 2D sketches has long been an open problem because the sketches only provide very sparse and ambiguous information. In this paper, we use an encoder/decoder architecture for the sketch to mesh translation. When integrated into a user interface that provides camera parameters for the sketches, this enables us to leverage its latent parametrization to represent and refine a 3D mesh so that its projections match the external contours outlined in the sketch. We will show that this approach is easy to deploy, robust to style changes, and effective. Furthermore, it can be used for shape refinement given only single pen strokes. We compare our approach to state-of-the-art methods on sketches -- both hand-drawn and synthesized -- and demonstrate that we outperform them.