# SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation

arXiv: https://arxiv.org/abs/2502.08642 · HTML: https://mlsanity.com/paper/2502.08642/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)
- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Style Aligned Image Generation via Shared Attention](https://mlsanity.com/paper/2312.02133/) (2023)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [SVGDreamer: Text Guided SVG Generation with Diffusion Model](https://mlsanity.com/paper/2312.16476/) (2023)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)
- [SketchyCOCO: Image Generation from Freehand Scene Sketches](https://mlsanity.com/paper/2003.02683/) (2020)
- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)
- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [Learning to Sketch with Shortcut Cycle Consistency](https://mlsanity.com/paper/1805.00247/) (2018)
- [Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt](https://mlsanity.com/paper/2005.09159/) (2020)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories](https://mlsanity.com/paper/1709.04121/) (2017)
- [VecFusion: Vector Font Generation with Diffusion](https://mlsanity.com/paper/2312.10540/) (2023)
- [NeuralSVG: An Implicit Representation for Text-to-Vector Generation](https://mlsanity.com/paper/2501.03992/) (2025)

## Built on by (most cited first, 14 of 14)

- [Jodi: Unification of Visual Generation and Understanding via Joint Modeling](https://mlsanity.com/paper/2505.19084/) (2025)
- [Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching](https://mlsanity.com/paper/2602.12280/) (2026)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback](https://mlsanity.com/paper/2510.20093/) (2025)
- [Birth of a Painting: Differentiable Brushstroke Reconstruction](https://mlsanity.com/paper/2511.13191/) (2025)
- [Teaching an Agent to Sketch One Part at a Time](https://mlsanity.com/paper/2603.19500/) (2026)
- [Vector sketch animation generation with differentiable motion trajectories](https://mlsanity.com/paper/2509.25857/) (2025)
- [O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model](https://mlsanity.com/paper/2511.14368/) (2025)
- [Order Matters: 3D Shape Generation from Sequential VR Sketches](https://mlsanity.com/paper/2512.04761/) (2025)
- [SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering](https://mlsanity.com/paper/2603.28363/) (2026)
- [A Few-Step Generative Model on Cumulative Flow Maps](https://mlsanity.com/paper/2605.03623/) (2026)
- [Single-Line Drawing Generation via Semantics-Driven Optimization](https://mlsanity.com/paper/2606.01910/) (2026)
- [GarmentSketch: Large-scale Sketch-to-Fashion Benchmark](https://mlsanity.com/paper/2606.14025/) (2026)
- [Draw This First](https://mlsanity.com/paper/2608.12064/) (2026)

## Abstract (arXiv)

Recent advancements in large vision-language models have enabled highly expressive and diverse vector sketch generation. However, state-of-the-art methods rely on a time-consuming optimization process involving repeated feedback from a pretrained model to determine stroke placement. Consequently, despite producing impressive sketches, these methods are limited in practical applications. In this work, we introduce SwiftSketch, a diffusion model for image-conditioned vector sketch generation that can produce high-quality sketches in less than a second. SwiftSketch operates by progressively denoising stroke control points sampled from a Gaussian distribution. Its transformer-decoder architecture is designed to effectively handle the discrete nature of vector representation and capture the inherent global dependencies between strokes. To train SwiftSketch, we construct a synthetic dataset of image-sketch pairs, addressing the limitations of existing sketch datasets, which are often created by non-artists and lack professional quality. For generating these synthetic sketches, we introduce ControlSketch, a method that enhances SDS-based techniques by incorporating precise spatial control through a depth-aware ControlNet. We demonstrate that SwiftSketch generalizes across diverse concepts, efficiently producing sketches that combine high fidelity with a natural and visually appealing style.