mlsanityData to 7 Oct 2026

Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model

arXiv 2208.12675 ↗First posted 26 Aug 2022cs.CVcited by 39

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Generating images from hand-drawings is a crucial and fundamental task in content creation. The translation is difficult as there exist infinite possibilities and the different users usually expect different outcomes. Therefore, we propose a unified framework supporting a three-dimensional control over the image synthesis from sketches and strokes based on diffusion models. Users can not only decide the level of faithfulness to the input strokes and sketches, but also the degree of realism, as the user inputs are usually not consistent with the real images. Qualitative and quantitative experiments demonstrate that our framework achieves state-of-the-art performance while providing flexibility in generating customized images with control over shape, color, and realism. Moreover, our method unleashes applications such as editing on real images, generation with partial sketches and strokes, and multi-domain multi-modal synthesis.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 23 of its 34 references that are in the map. First posted on arXiv 26 Aug 2022.

Builds on

23 of 34 references.

References · 23

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  9. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  10. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  11. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  12. 2017Toward Multimodal Image-to-Image Translationcited by 295
11 more
  1. 2019U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translationcited by 186
  2. 2018SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesiscited by 81
  3. 2016Scribbler: Controlling Deep Image Synthesis with Sketch and Colorcited by 77
  4. 2021Aligning Latent and Image Spaces to Connect the Unconnectablecited by 70
  5. 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 67
  6. 2021Paint Transformer: Feed Forward Neural Painting with Stroke Predictioncited by 61
  7. 2020Stylized Neural Paintingcited by 50
  8. 2019Unsupervised Sketch-to-Photo Synthesiscited by 38
  9. 2021Sketch Your Own GANcited by 37
  10. 2020Controllable Image Synthesis via SegVAEcited by 13
  11. 2020Self-Supervised Sketch-to-Image Synthesiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 16; 2024: 13; 2025: 6; 2026: 2.

39 papers.

Sort:

By area:

  1. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  2. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  3. 2023SKED: Sketch-guided Text-based 3D Editingcited by 72
  4. 2023Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editingcited by 62
  5. 2023DiffUTE: Universal Text Editing Diffusion Modelcited by 44
  6. 2022Diff-Font: Diffusion Model for Robust One-Shot Font Generationcited by 40
  7. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 37
  8. 2023Sketch and Text Guided Diffusion Model for Colored Point Cloud Generationcited by 36
  9. 2023Generating Visual Scenes from Touchcited by 26
  10. 2024Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Modelscited by 24
  11. 2023Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Modelscited by 17
  12. 2023Soundini: Sound-Guided Diffusion for Natural Video Editingcited by 14
27 more
  1. 2024Automated data processing and feature engineering for deep learning and big data applications: a surveycited by 14
  2. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  3. 2024Block and Detail: Scaffolding Sketch-to-Image Generationcited by 8
  4. 2023Realistic Data Enrichment for Robust Image Segmentation in Histopathologycited by 7
  5. 2024Training-Free Sketch-Guided Diffusion with Latent Optimizationcited by 7
  6. 2025LOTS of Fashion! Multi-Conditioning for Image Generation via Sketch-Text Pairingcited by 7
  7. 2023Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusioncited by 6
  8. 2024Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editingcited by 5
  9. 2024LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Modelcited by 5
  10. 2025Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstructioncited by 4
  11. 2024CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editingcited by 3
  12. 2024From Geometric Mimicry to Comprehensive Generation: A Context-Informed Multimodal Diffusion Model for Urban Morphology Synthesiscited by 3
  13. 2023Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Trainingcited by 2
  14. 2023Interactive Neural Paintingcited by 1
  15. 2023VIDES: Virtual Interior Design via Natural Language and Visual Guidancecited by 1
  16. 2024IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Imagescited by 1
  17. 2025Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Reviewcited by 1
  18. 2025Generative AI in Map-Making: A Technical Exploration and Its Implications for Cartographerscited by 1
  19. 2026SketchingReality: From Freehand Scene Sketches To Photorealistic Imagescited by 1
  20. 2023Text-Guided Scene Sketch-to-Photo Synthesiscited by 0
  21. 2023Customize StyleGAN with One Hand Sketchcited by 0
  22. 2024FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusioncited by 0
  23. 2024SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisationcited by 0
  24. 2024Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usagecited by 0
  25. 2024Urban Air Temperature Prediction using Conditional Diffusion Modelscited by 0
  26. 2025Layout Stroke Imitation: A Layout Guided Handwriting Stroke Generation for Style Imitation with Diffusion Modelcited by 0
  27. 2026Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VisioBlend: Sketch and Stroke-Guided Denoising Diffusion Probabilistic Model for Realistic Image Generation
  2. 2025CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model
  3. 2023SketchFFusion: Sketch-guided image editing with diffusion model
  4. 2023Conditional Human Sketch Synthesis with Explicit Abstraction Control
  5. 2025d-Sketch: Improving Visual Fidelity of Sketch-to-Image Translation with Pretrained Latent Diffusion Models without Retraining
  6. 2024KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models
  7. 2023Reference-based Image Composition with Sketch via Structure-aware Diffusion Model
  8. 2023DiffSketching: Sketch Control Image Synthesis with Diffusion Models