# Sketch Your Own GAN

arXiv: https://arxiv.org/abs/2108.02774 · HTML: https://mlsanity.com/paper/2108.02774/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 76 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 30 of 55)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)

## Built on by (most cited first, 30 of 37)

- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions](https://mlsanity.com/paper/2112.05219/) (2021)
- [FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context](https://mlsanity.com/paper/2203.02113/) (2022)
- [Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model](https://mlsanity.com/paper/2208.12675/) (2022)
- [Customizing Text-to-Image Models with a Single Image Pair](https://mlsanity.com/paper/2405.01536/) (2024)
- [State-of-the-Art in the Architecture, Methods and Applications of StyleGAN](https://mlsanity.com/paper/2202.14020/) (2022)
- [Picture that Sketch: Photorealistic Image Generation from Abstract Sketches](https://mlsanity.com/paper/2303.11162/) (2023)
- [SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches](https://mlsanity.com/paper/2111.15078/) (2021)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [MaLP: Manipulation Localization Using a Proactive Scheme](https://mlsanity.com/paper/2303.16976/) (2023)
- [Rewriting Geometric Rules of a GAN](https://mlsanity.com/paper/2207.14288/) (2022)
- [Facial-Sketch Synthesis: A New Challenge](https://mlsanity.com/paper/2112.15439/) (2021)
- [Domain Expansion of Image Generators](https://mlsanity.com/paper/2301.05225/) (2023)
- [DiffSketching: Sketch Control Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2305.18812/) (2023)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [MyStyle++: A Controllable Personalized Generative Prior](https://mlsanity.com/paper/2306.04865/) (2023)
- [DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing](https://mlsanity.com/paper/2109.10737/) (2021)
- [Content-Based Search for Deep Generative Models](https://mlsanity.com/paper/2210.03116/) (2022)
- [Extracting Semantic Knowledge from GANs with Unsupervised Learning](https://mlsanity.com/paper/2211.16710/) (2022)
- [Learning Geometry-aware Representations by Sketching](https://mlsanity.com/paper/2304.08204/) (2023)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2111.12757/) (2021)
- [Inversion-by-Inversion: Exemplar-based Sketch-to-Photo Synthesis via Stochastic Differential Equations without Training](https://mlsanity.com/paper/2308.07665/) (2023)
- [U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models](https://mlsanity.com/paper/2403.18425/) (2024)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [Intriguing Property and Counterfactual Explanation of GAN for Remote Sensing Image Generation](https://mlsanity.com/paper/2303.05240/) (2023)
- [Controlling Your Image via Simplified Vector Graphics](https://mlsanity.com/paper/2602.14443/) (2026)
- [SketchingReality: From Freehand Scene Sketches To Photorealistic Images](https://mlsanity.com/paper/2602.14648/) (2026)

## Abstract (arXiv)

Can a user create a deep generative model by sketching a single example? Traditionally, creating a GAN model has required the collection of a large-scale dataset of exemplars and specialized knowledge in deep learning. In contrast, sketching is possibly the most universally accessible way to convey a visual concept. In this work, we present a method, GAN Sketching, for rewriting GANs with one or more sketches, to make GANs training easier for novice users. In particular, we change the weights of an original GAN model according to user sketches. We encourage the model's output to match the user sketches through a cross-domain adversarial loss. Furthermore, we explore different regularization methods to preserve the original model's diversity and image quality. Experiments have shown that our method can mold GANs to match shapes and poses specified by sketches while maintaining realism and diversity. Finally, we demonstrate a few applications of the resulting GAN, including latent space interpolation and image editing.