mlsanityData to 14 Oct 2026

SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisation

arXiv 2405.18716 ↗First posted 29 May 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce SketchDeco, a training-free approach to sketch colourisation that bridges the gap between professional design needs and intuitive, region-based control. Our method empowers artists to use simple masks and colour palettes for precise spatial and chromatic specification, avoiding both the tediousness of manual assignment and the ambiguity of text-based prompts. We reformulate this task as a novel, training-free composition problem. Our core technical contribution is a guided latent-space blending process: we first leverage diffusion inversion to precisely ``paint'' user-defined colours into specified regions, and then use a custom self-attention mechanism to harmoniously blend these local edits with a globally consistent base image. This ensures both local colour fidelity and global harmony without requiring any model fine-tuning. Our system produces high-quality results in 15--20 inference steps on consumer GPUs, making professional-quality, controllable colourisation accessible.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 62 of its 86 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

62 of 86 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  8. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  9. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  11. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  12. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  13. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  14. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  15. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  17. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  18. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  19. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  20. 2023FreeU: Free Lunch in Diffusion U-Netcited by 196
  21. 2019U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translationcited by 186
  22. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  23. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  24. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 168
  25. 2017Real-Time User-Guided Image Colorization with Learned Deep Priorscited by 123
  26. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 91
  27. 2018SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesiscited by 81
  28. 2016Scribbler: Controlling Deep Image Synthesis with Sketch and Colorcited by 78
  29. 2021Towards Vivid and Diverse Image Colorization with Generative Color Priorcited by 55
  30. 2017Language-Based Image Editing with Recurrent Attentive Modelscited by 53
  31. 2020Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondencecited by 53
  32. 2023CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulationcited by 41
  33. 2022Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Modelcited by 39
  34. 2019Unsupervised Sketch-to-Photo Synthesiscited by 38
  35. 2021Sketch Your Own GANcited by 37
  36. 2018Coloring with Words: Guiding Image Colorization Through Text-based Palette Generationcited by 35
  37. 2017Image Generation from Sketch Constraint Using Contextual GANcited by 34
  38. 2022Sketch3T: Test-Time Training for Zero-Shot SBIRcited by 34

50 most cited of 62 All 62 (JSON)

Built on by

Not cited yet.