# Unsupervised Sketch-to-Photo Synthesis

arXiv: https://arxiv.org/abs/1909.08313 · HTML: https://mlsanity.com/paper/1909.08313/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 30 of 38)

- [DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models](https://mlsanity.com/paper/2306.14685/) (2023)
- [Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model](https://mlsanity.com/paper/2208.12675/) (2022)
- [Picture that Sketch: Photorealistic Image Generation from Abstract Sketches](https://mlsanity.com/paper/2303.11162/) (2023)
- [It's All About Your Sketch: Democratising Sketch Control in Diffusion Models](https://mlsanity.com/paper/2403.07234/) (2024)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation](https://mlsanity.com/paper/2102.12593/) (2021)
- [Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2104.05703/) (2021)
- [SketchLattice: Latticed Representation for Sketch Manipulation](https://mlsanity.com/paper/2108.11636/) (2021)
- [Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings](https://mlsanity.com/paper/2303.11502/) (2023)
- [DiffSketching: Sketch Control Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2305.18812/) (2023)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [One Sketch for All: One-Shot Personalized Sketch Segmentation](https://mlsanity.com/paper/2112.10838/) (2021)
- [Block and Detail: Scaffolding Sketch-to-Image Generation](https://mlsanity.com/paper/2402.18116/) (2024)
- [Deep Image Synthesis from Intuitive User Input: A Review and Perspectives](https://mlsanity.com/paper/2107.04240/) (2021)
- [Freehand Sketch Generation from Mechanical Components](https://mlsanity.com/paper/2408.05966/) (2024)
- [Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing](https://mlsanity.com/paper/2208.08092/) (2022)
- [DAVINCI: A Single-Stage Architecture for Constrained CAD Sketch Inference](https://mlsanity.com/paper/2410.22857/) (2024)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [Unsupervised Scene Sketch to Photo Synthesis](https://mlsanity.com/paper/2209.02834/) (2022)
- [ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2111.12757/) (2021)
- [Improving Sketch Colorization using Adversarial Segmentation Consistency](https://mlsanity.com/paper/2301.08590/) (2023)
- [CraftSVG: Multi-Object Text-to-SVG Synthesis via Layout Guided Diffusion](https://mlsanity.com/paper/2404.00412/) (2024)
- [Locally-Focused Face Representation for Sketch-to-Image Generation Using Noise-Induced Refinement](https://mlsanity.com/paper/2411.19005/) (2024)
- [Adversarial Segmentation Loss for Sketch Colorization](https://mlsanity.com/paper/2102.06192/) (2021)
- [Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training](https://mlsanity.com/paper/2308.07665/) (2023)
- [U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models](https://mlsanity.com/paper/2403.18425/) (2024)
- [Adversarial Open Domain Adaption Framework (AODA): Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2108.04351/) (2021)
- [Style-Content Disentanglement in Language-Image Pretraining Representations for Zero-Shot Sketch-to-Image Synthesis](https://mlsanity.com/paper/2206.01661/) (2022)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)
- [HAIFIT: Human-to-AI Fashion Image Translation](https://mlsanity.com/paper/2403.08651/) (2024)

## Abstract (arXiv)

Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.