mlsanityData to 14 Oct 2026

CoGS: Controllable Generation and Search from Sketch and Style

arXiv 2203.09554 ↗First posted 17 Mar 2022cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

We present CoGS, a novel method for the style-conditioned, sketch-driven synthesis of images. CoGS enables exploration of diverse appearance possibilities for a given sketched object, enabling decoupled control over the structure and the appearance of the output. Coarse-grained control over object structure and appearance are enabled via an input sketch and an exemplar "style" conditioning image to a transformer-based sketch and style encoder to generate a discrete codebook representation. We map the codebook representation into a metric space, enabling fine-grained control over selection and interpolation between multiple synthesis options before generating the image via a vector quantized GAN (VQGAN) decoder. Our framework thereby unifies search and synthesis tasks, in that a sketch and style pair may be used to run an initial synthesis which may be refined via combination with similar results in a search corpus to produce an image more closely matching the user's intent. We show that our model, trained on the 125 object classes of our newly created Pseudosketches dataset, is capable of producing a diverse gamut of semantic content and appearance styles.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

41 of 72 references.

References · 41

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2014Conditional Generative Adversarial Netscited by 2,162
  10. 2019Contrastive Multiview Codingcited by 1,301
  11. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
29 more
  1. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  3. 2016Generative Adversarial Text to Image Synthesiscited by 735
  4. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  5. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  6. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  7. 2021Instance-Conditioned GANcited by 121
  8. 2016Learning What and Where to Drawcited by 105
  9. 2018Image Generation from Layoutcited by 105
  10. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  11. 2018SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesiscited by 81
  12. 2016Scribbler: Controlling Deep Image Synthesis with Sketch and Colorcited by 78
  13. 2020Sketchformer: Transformer-based Representation for Sketched Structurecited by 78
  14. 2019AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networkscited by 77
  15. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  16. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  17. 2020SketchyCOCO: Image Generation from Freehand Scene Sketchescited by 67
  18. 2021Image Inpainting via Conditional Texture and Structure Dual Generationcited by 66
  19. 2020Object-Centric Image Generation from Layoutscited by 57
  20. 2019Interactive Sketch & Fill: Multiclass Sketch-to-Image Translationcited by 51
  21. 2017TextureGAN: Controlling Deep Image Synthesis with Texture Patchescited by 47
  22. 2021Exemplar-Based Open-Set Panoptic Segmentation Networkcited by 38
  23. 2017Image Generation from Sketch Constraint Using Contextual GANcited by 34
  24. 2021ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similaritycited by 34
  25. 2021Deep Image Synthesis from Intuitive User Input: A Review and Perspectivescited by 8
  26. 2020S2FGAN: Semantically Aware Interactive Sketch-to-Face Translationcited by 6
  27. 2016Convolutional Sketch Inversioncited by 4
  28. 2020CoCosNet v2: Full-Resolution Correspondence Learning for Image Translationcited by 4
  29. 2021Scene Designer: a Unified Model for Scene Search and Synthesis from Sketchcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 5; 2025: 1; 2026: 1.

14 papers.

Sort:

By area:

  1. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  2. 2023Picture that Sketch: Photorealistic Image Generation from Abstract Sketchescited by 34
  3. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 29
  4. 2023MaskSketch: Unpaired Structure-guided Masked Image Generationcited by 27
  5. 2023Modulating Pretrained Diffusion Models for Multimodal Image Synthesiscited by 23
  6. 2023What Can Human Sketches Do for Object Detection?cited by 19
  7. 2024You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrievalcited by 12
  8. 2023PARASOL: Parametric Style Control for Diffusion Image Synthesiscited by 5
  9. 2024LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Modelcited by 5
  10. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  11. 2023ALADIN-NST: Self-supervised disentangled representation learning of artistic style through Neural Style Transfercited by 3
  12. 2024High-Quality Medical Image Generation from Free-hand Sketchcited by 2
2 more
  1. 2024SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisationcited by 0
  2. 2026Diff-SBSR: Learning Multimodal Feature-Enhanced Diffusion Models for Zero-Shot Sketch-Based 3D Shape Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Customize StyleGAN with One Hand Sketch
  2. 2023Conditional Human Sketch Synthesis with Explicit Abstraction Control
  3. 2025Text to Sketch Generation with Multi-Styles
  4. 2023DiffSketching: Sketch Control Image Synthesis with Diffusion Models
  5. 2025d-Sketch: Improving Visual Fidelity of Sketch-to-Image Translation with Pretrained Latent Diffusion Models without Retraining
  6. 2026Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation
  7. 2025SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation
  8. 2024KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models