# Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis

arXiv: https://arxiv.org/abs/2104.05703 · HTML: https://mlsanity.com/paper/2104.05703/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 84 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 56)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks](https://mlsanity.com/paper/1604.04382/) (2016)
- [U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation](https://mlsanity.com/paper/1907.10830/) (2019)
- [SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color](https://mlsanity.com/paper/1902.06838/) (2019)
- [Semi-Supervised Learning with Generative Adversarial Networks](https://mlsanity.com/paper/1606.01583/) (2016)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1904.03451/) (2019)

## Built on by (most cited first, 24 of 24)

- [FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model](https://mlsanity.com/paper/2303.09833/) (2023)
- [Sketch-Guided Text-to-Image Diffusion Models](https://mlsanity.com/paper/2211.13752/) (2022)
- [Picture that Sketch: Photorealistic Image Generation from Abstract Sketches](https://mlsanity.com/paper/2303.11162/) (2023)
- [Understanding and Improving Training-free Loss-based Diffusion Guidance](https://mlsanity.com/paper/2403.12404/) (2024)
- [Decoupling Training-Free Guided Diffusion by ADMM](https://mlsanity.com/paper/2411.12773/) (2024)
- [ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text](https://mlsanity.com/paper/2401.01456/) (2024)
- [ToonComposer: Streamlining Cartoon Production with Generative Post-Keyframing](https://mlsanity.com/paper/2508.10881/) (2025)
- [Sketch-guided Image Inpainting with Partial Discrete Diffusion Process](https://mlsanity.com/paper/2404.11949/) (2024)
- [Training-Free Sketch-Guided Diffusion with Latent Optimization](https://mlsanity.com/paper/2409.00313/) (2024)
- [Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing](https://mlsanity.com/paper/2208.08092/) (2022)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [Image Referenced Sketch Colorization Based on Animation Creation Workflow](https://mlsanity.com/paper/2502.19937/) (2025)
- [Unsupervised Scene Sketch to Photo Synthesis](https://mlsanity.com/paper/2209.02834/) (2022)
- [IT-RUDA: Information Theory Assisted Robust Unsupervised Domain Adaptation](https://mlsanity.com/paper/2210.12947/) (2022)
- [MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention](https://mlsanity.com/paper/2501.00816/) (2025)
- [Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training](https://mlsanity.com/paper/2308.07665/) (2023)
- [Test-time Conditional Text-to-Image Synthesis Using Diffusion Models](https://mlsanity.com/paper/2411.10800/) (2024)
- [Style-Content Disentanglement in Language-Image Pretraining Representations for Zero-Shot Sketch-to-Image Synthesis](https://mlsanity.com/paper/2206.01661/) (2022)
- [DiffRGD: An Inference-Time Diffusion Guidance Through Riemannian Gradient Descent](https://mlsanity.com/paper/2606.28417/) (2026)
- [Text-Guided Scene Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2302.06883/) (2023)
- [Flow-Guided Controllable Line Drawing Generation](https://mlsanity.com/paper/2307.07540/) (2023)
- [Learning to Synthesize Graphics Programs for Geometric Artworks](https://mlsanity.com/paper/2410.15768/) (2024)
- [Uni-Animator: Towards Unified Visual Colorization](https://mlsanity.com/paper/2602.23191/) (2026)
- [Towards High-resolution and Disentangled Reference-based Sketch Colorization](https://mlsanity.com/paper/2603.05971/) (2026)

## Abstract (arXiv)

In this paper, we explore open-domain sketch-to-photo translation, which aims to synthesize a realistic photo from a freehand sketch with its class label, even if the sketches of that class are missing in the training data. It is challenging due to the lack of training supervision and the large geometric distortion between the freehand sketch and photo domains. To synthesize the absent freehand sketches from photos, we propose a framework that jointly learns sketch-to-photo and photo-to-sketch generation. However, the generator trained from fake sketches might lead to unsatisfying results when dealing with sketches of missing classes, due to the domain gap between synthesized sketches and real ones. To alleviate this issue, we further propose a simple yet effective open-domain sampling and optimization strategy to "fool" the generator into treating fake sketches as real ones. Our method takes advantage of the learned sketch-to-photo and photo-to-sketch mapping of in-domain data and generalizes it to the open-domain classes. We validate our method on the Scribble and SketchyCOCO datasets. Compared with the recent competing methods, our approach shows impressive results in synthesizing realistic color, texture, and maintaining the geometric composition for various categories of open-domain sketches. Our code is available at https://github.com/Mukosame/AODA