# Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model

arXiv: https://arxiv.org/abs/2208.12675 · HTML: https://mlsanity.com/paper/2208.12675/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation](https://mlsanity.com/paper/1907.10830/) (2019)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Aligning Latent and Image Spaces to Connect the Unconnectable](https://mlsanity.com/paper/2104.06954/) (2021)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)
- [Paint Transformer: Feed Forward Neural Painting with Stroke Prediction](https://mlsanity.com/paper/2108.03798/) (2021)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Unsupervised Sketch-to-Photo Synthesis](https://mlsanity.com/paper/1909.08313/) (2019)
- [Sketch Your Own GAN](https://mlsanity.com/paper/2108.02774/) (2021)
- [Controllable Image Synthesis via SegVAE](https://mlsanity.com/paper/2007.08397/) (2020)
- [Self-Supervised Sketch-to-Image Synthesis](https://mlsanity.com/paper/2012.09290/) (2020)

## Built on by (most cited first, 30 of 39)

- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [SKED: Sketch-guided Text-based 3D Editing](https://mlsanity.com/paper/2303.10735/) (2023)
- [Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing](https://mlsanity.com/paper/2304.02051/) (2023)
- [DiffUTE: Universal Text Editing Diffusion Model](https://mlsanity.com/paper/2305.10825/) (2023)
- [Diff-Font: Diffusion Model for Robust One-Shot Font Generation](https://mlsanity.com/paper/2212.05895/) (2022)
- [Exploiting Diffusion Prior for Generalizable Dense Prediction](https://mlsanity.com/paper/2311.18832/) (2023)
- [Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation](https://mlsanity.com/paper/2308.02874/) (2023)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models](https://mlsanity.com/paper/2405.09901/) (2024)
- [Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models](https://mlsanity.com/paper/2304.01182/) (2023)
- [Soundini: Sound-Guided Diffusion for Natural Video Editing](https://mlsanity.com/paper/2304.06818/) (2023)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [CoDe: Blockwise Control for Denoising Diffusion Models](https://mlsanity.com/paper/2502.00968/) (2025)
- [Block and Detail: Scaffolding Sketch-to-Image Generation](https://mlsanity.com/paper/2402.18116/) (2024)
- [Realistic Data Enrichment for Robust Image Segmentation in Histopathology](https://mlsanity.com/paper/2304.09534/) (2023)
- [Training-Free Sketch-Guided Diffusion with Latent Optimization](https://mlsanity.com/paper/2409.00313/) (2024)
- [LOTS of Fashion! Multi-Conditioning for Image Generation via Sketch-Text Pairing](https://mlsanity.com/paper/2507.22627/) (2025)
- [Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion](https://mlsanity.com/paper/2301.13743/) (2023)
- [Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing](https://mlsanity.com/paper/2403.14828/) (2024)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction](https://mlsanity.com/paper/2505.13091/) (2025)
- [CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing](https://mlsanity.com/paper/2402.17624/) (2024)
- [From Geometric Mimicry to Comprehensive Generation: A Context-Informed Multimodal Diffusion Model for Urban Morphology Synthesis](https://mlsanity.com/paper/2409.17049/) (2024)
- [Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training](https://mlsanity.com/paper/2308.07665/) (2023)
- [Interactive Neural Painting](https://mlsanity.com/paper/2307.16441/) (2023)
- [VIDES: Virtual Interior Design via Natural Language and Visual Guidance](https://mlsanity.com/paper/2308.13795/) (2023)
- [IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images](https://mlsanity.com/paper/2404.00269/) (2024)
- [Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2505.07866/) (2025)
- [Generative AI in Map-Making: A Technical Exploration and Its Implications for Cartographers](https://mlsanity.com/paper/2508.18959/) (2025)

## Abstract (arXiv)

Generating images from hand-drawings is a crucial and fundamental task in content creation. The translation is difficult as there exist infinite possibilities and the different users usually expect different outcomes. Therefore, we propose a unified framework supporting a three-dimensional control over the image synthesis from sketches and strokes based on diffusion models. Users can not only decide the level of faithfulness to the input strokes and sketches, but also the degree of realism, as the user inputs are usually not consistent with the real images. Qualitative and quantitative experiments demonstrate that our framework achieves state-of-the-art performance while providing flexibility in generating customized images with control over shape, color, and realism. Moreover, our method unleashes applications such as editing on real images, generation with partial sketches and strokes, and multi-domain multi-modal synthesis.