# SketchFaceGS: Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting

arXiv: https://arxiv.org/abs/2604.19202 · HTML: https://mlsanity.com/paper/2604.19202/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [Escaping Plato's Cave: 3D Shape From Adversarial Rendering](https://mlsanity.com/paper/1811.11606/) (2018)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [3D Shape Induction from 2D Views of Multiple Objects](https://mlsanity.com/paper/1612.05872/) (2016)
- [GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing](https://mlsanity.com/paper/2403.08733/) (2024)
- [SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color](https://mlsanity.com/paper/1902.06838/) (2019)
- [Generative Neural Articulated Radiance Fields](https://mlsanity.com/paper/2206.14314/) (2022)
- [Generalizable and Animatable Gaussian Head Avatar](https://mlsanity.com/paper/2410.07971/) (2024)
- [GIRAFFE HD: A High-Resolution 3D-aware Generative Model](https://mlsanity.com/paper/2203.14954/) (2022)
- [3D Gaussian Blendshapes for Head Avatar Animation](https://mlsanity.com/paper/2404.19398/) (2024)
- [HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors](https://mlsanity.com/paper/2406.12459/) (2024)
- [GGHead: Fast and Generalizable 3D Gaussian Heads](https://mlsanity.com/paper/2406.09377/) (2024)
- [Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches](https://mlsanity.com/paper/2001.02890/) (2020)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)

## Abstract (arXiv)

3D Gaussian representations have emerged as a powerful paradigm for digital head modeling, achieving photorealistic quality with real-time rendering. However, intuitive and interactive creation or editing of 3D Gaussian head models remains challenging. Although 2D sketches provide an ideal interaction modality for fast, intuitive conceptual design, they are sparse, depth-ambiguous, and lack high-frequency appearance cues, making it difficult to infer dense, geometrically consistent 3D Gaussian structures from strokes - especially under real-time constraints. To address these challenges, we propose SketchFaceGS, the first sketch-driven framework for real-time generation and editing of photorealistic 3D Gaussian head models from 2D sketches. Our method uses a feed-forward, coarse-to-fine architecture. A Transformer-based UV feature-prediction module first reconstructs a coarse but geometrically consistent UV feature map from the input sketch, and then a 3D UV feature enhancement module refines it with high-frequency, photorealistic detail to produce a high-fidelity 3D head. For editing, we introduce a UV Mask Fusion technique combined with a layer-by-layer feature-fusion strategy, enabling precise, real-time, free-viewpoint modifications. Extensive experiments show that SketchFaceGS outperforms existing methods in both generation fidelity and editing flexibility, producing high-quality, editable 3D heads from sketches in a single forward pass.