# DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models

arXiv: https://arxiv.org/abs/2306.14685 · HTML: https://mlsanity.com/paper/2306.14685/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 39 of its 47 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models](https://mlsanity.com/paper/2211.11319/) (2022)
- [Learning to generate line drawings that convey geometry and semantics](https://mlsanity.com/paper/2203.12691/) (2022)
- [Towards Layer-wise Image Vectorization](https://mlsanity.com/paper/2206.04655/) (2022)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)
- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)
- [Creative Sketch Generation](https://mlsanity.com/paper/2011.10039/) (2020)
- [Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes](https://mlsanity.com/paper/2103.17185/) (2021)

## Built on by (most cited first, 30 of 54)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter](https://mlsanity.com/paper/2309.02773/) (2023)
- [SVGDreamer: Text Guided SVG Generation with Diffusion Model](https://mlsanity.com/paper/2312.16476/) (2023)
- [Empowering LLMs to Understand and Generate Complex Vector Graphics](https://mlsanity.com/paper/2412.11102/) (2024)
- [Posterior Distillation Sampling](https://mlsanity.com/paper/2311.13831/) (2023)
- [COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design](https://mlsanity.com/paper/2311.16974/) (2023)
- [Rethinking Score Distillation as a Bridge Between Image Distributions](https://mlsanity.com/paper/2406.09417/) (2024)
- [Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models](https://mlsanity.com/paper/2411.16602/) (2024)
- [SketchAgent: Language-Driven Sequential Sketch Generation](https://mlsanity.com/paper/2411.17673/) (2024)
- [NeuralSVG: An Implicit Representation for Text-to-Vector Generation](https://mlsanity.com/paper/2501.03992/) (2025)
- [Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities](https://mlsanity.com/paper/2406.14562/) (2024)
- [SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation](https://mlsanity.com/paper/2506.03139/) (2025)
- [SVGen: Interpretable Vector Graphics Generation with Large Language Models](https://mlsanity.com/paper/2508.09168/) (2025)
- [SketchINR: A First Look into Sketches as Implicit Neural Representations](https://mlsanity.com/paper/2403.09344/) (2024)
- [U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers](https://mlsanity.com/paper/2503.08157/) (2025)
- [Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback](https://mlsanity.com/paper/2604.20730/) (2026)
- [Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering](https://mlsanity.com/paper/2405.15305/) (2024)
- [OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typography](https://mlsanity.com/paper/2506.21101/) (2025)
- [DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance](https://mlsanity.com/paper/2512.10894/) (2025)
- [VectorPainter: Advanced Stylized Vector Graphics Synthesis Using Stroke-Style Priors](https://mlsanity.com/paper/2405.02962/) (2024)
- [Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints](https://mlsanity.com/paper/2411.19381/) (2024)
- [VAnim: Rendering-Aware Sparse State Modeling for Structure-Preserving Vector Animation](https://mlsanity.com/paper/2605.01517/) (2026)
- [Bezier Splatting for Fast and Differentiable Vector Graphics Rendering](https://mlsanity.com/paper/2503.16424/) (2025)
- [ViewCraft3D: High-Fidelity and View-Consistent 3D Vector Graphics Synthesis](https://mlsanity.com/paper/2505.19492/) (2025)
- [RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling](https://mlsanity.com/paper/2503.09601/) (2025)
- [IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework](https://mlsanity.com/paper/2603.09312/) (2026)
- [CraftSVG: Multi-Object Text-to-SVG Synthesis via Layout Guided Diffusion](https://mlsanity.com/paper/2404.00412/) (2024)
- [SVGFusion: A VAE-Diffusion Transformer for Vector Graphic Generation](https://mlsanity.com/paper/2412.10437/) (2024)
- [Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching](https://mlsanity.com/paper/2602.12280/) (2026)
- [AmodalSVG: Amodal Image Vectorization via Semantic Layer Peeling](https://mlsanity.com/paper/2604.10940/) (2026)

## Abstract (arXiv)

We demonstrate that pre-trained text-to-image diffusion models, despite being trained on raster images, possess a remarkable capacity to guide vector sketch synthesis. In this paper, we introduce DiffSketcher, a novel algorithm for generating vectorized free-hand sketches directly from natural language prompts. Our method optimizes a set of B'ezier curves via an extended Score Distillation Sampling (SDS) loss, successfully bridging a raster-level diffusion prior with a parametric vector generator. To further accelerate the generation process, we propose a stroke initialization strategy driven by the diffusion model's intrinsic attention maps. Results show that DiffSketcher produces sketches across varying levels of abstraction while maintaining the structural integrity and essential visual details of the subject. Experiments confirm that our approach yields superior perceptual quality and controllability over existing methods. The code and demo are available at https://ximinng.github.io/DiffSketcher-project/