mlsanityData to 14 Oct 2026

It's All About Your Sketch: Democratising Sketch Control in Diffusion Models

arXiv 2403.07234 ↗First posted 12 Mar 2024cs.CVcited by 29

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get". A pilot study underscores the necessity, revealing that deformities in existing models stem from spatial-conditioning. To rectify this, we propose an abstraction-aware framework, utilising a sketch adapter, adaptive time-step sampling, and discriminative guidance from a pre-trained fine-grained sketch-based image retrieval model, working synergistically to reinforce fine-grained sketch-photo association. Our approach operates seamlessly during inference without the need for textual prompts; a simple, rough sketch akin to what you and I can create suffices! We welcome everyone to examine results presented in the paper and its supplementary. Contributions include democratising sketch control, introducing an abstraction-aware framework, and leveraging discriminative guidance, validated through extensive experiments.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 72 of its 90 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

72 of 90 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Emergent Correspondence from Image Diffusion6 Jun
  2. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editing20 Apr
  3. 2023Discriminative Class Tokens for Text-to-Image Diffusion Models30 Mar
  4. 2023Your Diffusion Model is Secretly a Zero-Shot Classifier28 Mar
  5. 2023ReVersion: Diffusion-Based Relation Inversion from Images23 Mar
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  8. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  9. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  11. 2022Diffusion-based Image Translation using Disentangled Style and Content Representation30 Sep
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
4 more
  1. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  5. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  6. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  7. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  9. 2015Holistically-Nested Edge Detectioncited by 592
  10. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  11. 2021MagFace: A Universal Representation for Face Recognition and Quality Assessmentcited by 334
  12. 2017A Neural Representation of Sketch Drawingscited by 313
  13. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 213
  14. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 168
  15. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 156
  16. 2021Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Imagescited by 113
  17. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  18. 2022Learning to generate line drawings that convey geometry and semanticscited by 83
  19. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  20. 2023SKED: Sketch-guided Text-based 3D Editingcited by 73
  21. 2019Interactive Sketch & Fill: Multiclass Sketch-to-Image Translationcited by 51
  22. 2021StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrievalcited by 50
  23. 2019Unsupervised Sketch-to-Photo Synthesiscited by 38
  24. 2021More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrievalcited by 38
  25. 2017Image Generation from Sketch Constraint Using Contextual GANcited by 34
  26. 2023Picture that Sketch: Photorealistic Image Generation from Abstract Sketchescited by 34
  27. 2022SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Textcited by 30
  28. 2020Why Do Line Drawings Work? A Realism Hypothesiscited by 28
  29. 2022A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketchcited by 28
  30. 2022Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrievalcited by 27
  31. 2022Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketchescited by 23
  32. 2022Partially Does It: Towards Scene-Level FG-SBIR with Partial Inputcited by 20
  33. 2023A Method for Animating Children's Drawings of the Human Figurecited by 20
  34. 2023Medical diffusion on a budget: Textual Inversion for medical image generationcited by 20
  35. 2023What Can Human Sketches Do for Object Detection?cited by 19
  36. 2023Sketch2Saliency: Learning to Detect Salient Objects from Human Drawingscited by 17
  37. 2022CoGS: Controllable Generation and Search from Sketch and Stylecited by 14
  38. 2023Exploiting Unlabelled Photos for Stronger Fine-Grained SBIRcited by 14

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 10.

29 papers.

Sort:

By area:

  1. 2025IMAGGarment: Fine-Grained Garment Generation for Controllable Fashion Designcited by 36
  2. 2025HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignmentcited by 10
  3. 2025VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Controlcited by 9
  4. 2024FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animationscited by 8
  5. 2024Training-Free Sketch-Guided Diffusion with Latent Optimizationcited by 7
  6. 2024KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Modelscited by 7
  7. 2025LOTS of Fashion! Multi-Conditioning for Image Generation via Sketch-Text Pairingcited by 7
  8. 2024LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Modelcited by 5
  9. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  10. 2025Diff-Palm: Realistic Palmprint Generation with Polynomial Creases and Intra-Class Variation Controllable Diffusion Modelscited by 3
  11. 2024U-Sketch: An Efficient Approach for Sketch to Image Diffusion Modelscited by 2
  12. 2024I2AM: Interpreting Image-to-Image Latent Diffusion Models via Bi-Attribution Mapscited by 2
17 more
  1. 2025LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMscited by 2
  2. 2025SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketchescited by 1
  3. 2025Universal Multi-Domain Translation via Diffusion Routerscited by 1
  4. 2025StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedbackcited by 1
  5. 2026SketchingReality: From Freehand Scene Sketches To Photorealistic Imagescited by 1
  6. 2026LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulationcited by 1
  7. 2024Multimodal Conditional 3D Face Geometry Generationcited by 0
  8. 2025StructDiff: Structure-aware Diffusion Model for 3D Fine-grained Medical Image Synthesiscited by 0
  9. 2025O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Modelcited by 0
  10. 2026Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generationcited by 0
  11. 2026Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusioncited by 0
  12. 2026SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answeringcited by 0
  13. 2026FlowPalm: Optical Flow Driven Non-Rigid Deformation for Geometrically Diverse Palmprint Generationcited by 0
  14. 2026Geometrically Consistent Multi-View Scene Generation from Freehand Sketchescited by 0
  15. 2026GarmentSketch: Large-scale Sketch-to-Fashion Benchmarkcited by 0
  16. 2026InstanceControl: Controllable Complex Image Generation without Instance Labelingcited by 0
  17. 2026ProgressNet: Sketching and Prompting with a Frozen Text-to-Image Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Conditional Human Sketch Synthesis with Explicit Abstraction Control
  2. 2023DiffSketching: Sketch Control Image Synthesis with Diffusion Models
  3. 2025CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model
  4. 2023SketchFFusion: Sketch-guided image editing with diffusion model
  5. 2025SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation
  6. 2025Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch
  7. 2023SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation
  8. 2025d-Sketch: Improving Visual Fidelity of Sketch-to-Image Translation with Pretrained Latent Diffusion Models without Retraining