# DiffSketching: Sketch Control Image Synthesis with Diffusion Models

arXiv: https://arxiv.org/abs/2305.18812 · HTML: https://mlsanity.com/paper/2305.18812/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 30 May 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Fast Sampling of Diffusion Models with Exponential Integrator](https://mlsanity.com/paper/2204.13902/) (2022)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Flexible Diffusion Modeling of Long Videos](https://mlsanity.com/paper/2205.11495/) (2022)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Conditional Image Generation with Score-Based Diffusion Models](https://mlsanity.com/paper/2111.13606/) (2021)
- [Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality](https://mlsanity.com/paper/2202.05830/) (2022)
- [On Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.00132/) (2021)
- [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022)
- [DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents](https://mlsanity.com/paper/2201.00308/) (2022)
- [Accelerating Diffusion Models via Early Stop of the Diffusion Process](https://mlsanity.com/paper/2205.12524/) (2022)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)

## Built on by (most cited first, 10 of 10)

- [Group Diffusion Transformers are Unsupervised Multitask Learners](https://mlsanity.com/paper/2410.15027/) (2024)
- [Terrain Diffusion Network: Climatic-Aware Terrain Generation with Geological Sketch Guidance](https://mlsanity.com/paper/2308.16725/) (2023)
- [CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing](https://mlsanity.com/paper/2402.17624/) (2024)
- [Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training](https://mlsanity.com/paper/2308.07665/) (2023)
- [U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models](https://mlsanity.com/paper/2403.18425/) (2024)
- [SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation](https://mlsanity.com/paper/2405.18801/) (2024)
- [StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback](https://mlsanity.com/paper/2510.20093/) (2025)
- [SketchingReality: From Freehand Scene Sketches To Photorealistic Images](https://mlsanity.com/paper/2602.14648/) (2026)
- [Multiple Noises in Diffusion Model for Semi-Supervised Multi-Domain Translation](https://mlsanity.com/paper/2309.14394/) (2023)
- [SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisation](https://mlsanity.com/paper/2405.18716/) (2024)

## Abstract (arXiv)

Creative sketch is a universal way of visual expression, but translating images from an abstract sketch is very challenging. Traditionally, creating a deep learning model for sketch-to-image synthesis needs to overcome the distorted input sketch without visual details, and requires to collect large-scale sketch-image datasets. We first study this task by using diffusion models. Our model matches sketches through the cross domain constraints, and uses a classifier to guide the image synthesis more accurately. Extensive experiments confirmed that our method can not only be faithful to user's input sketches, but also maintain the diversity and imagination of synthetic image results. Our model can beat GAN-based method in terms of generation quality and human evaluation, and does not rely on massive sketch-image datasets. Additionally, we present applications of our method in image editing and interpolation.