# S3D: Sketch-Driven 3D Model Generation

arXiv: https://arxiv.org/abs/2505.04185 · HTML: https://mlsanity.com/paper/2505.04185/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 21 of 21)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization](https://mlsanity.com/paper/2303.07418/) (2023)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [Escaping Plato's Cave: 3D Shape From Adversarial Rendering](https://mlsanity.com/paper/1811.11606/) (2018)
- [PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360circ](https://mlsanity.com/paper/2303.13071/) (2023)
- [VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment](https://mlsanity.com/paper/2312.04651/) (2023)
- [3D-aware Conditional Image Synthesis](https://mlsanity.com/paper/2302.08509/) (2023)
- [Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation](https://mlsanity.com/paper/2303.09036/) (2023)
- [Progressive Text-to-3D Generation for Automatic 3D Prototyping](https://mlsanity.com/paper/2309.14600/) (2023)

## Built on by (most cited first, 1 of 1)

- [SketchFaceGS: Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting](https://mlsanity.com/paper/2604.19202/) (2026)

## Abstract (arXiv)

Generating high-quality 3D models from 2D sketches is a challenging task due to the inherent ambiguity and sparsity of sketch data. In this paper, we present S3D, a novel framework that converts simple hand-drawn sketches into detailed 3D models. Our method utilizes a U-Net-based encoder-decoder architecture to convert sketches into face segmentation masks, which are then used to generate a 3D representation that can be rendered from novel views. To ensure robust consistency between the sketch domain and the 3D output, we introduce a novel style-alignment loss that aligns the U-Net bottleneck features with the initial encoder outputs of the 3D generation module, significantly enhancing reconstruction fidelity. To further enhance the network's robustness, we apply augmentation techniques to the sketch dataset. This streamlined framework demonstrates the effectiveness of S3D in generating high-quality 3D models from sketch inputs. The source code for this project is publicly available at https://github.com/hailsong/S3D.