# DoodleFormer: Creative Sketch Drawing with Transformers

arXiv: https://arxiv.org/abs/2112.03258 · HTML: https://mlsanity.com/paper/2112.03258/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generation](https://mlsanity.com/paper/2003.06988/) (2020)
- [Image Synthesis From Reconfigurable Layout and Style](https://mlsanity.com/paper/1908.07500/) (2019)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [StyleGAN2 Distillation for Feed-forward Image Manipulation](https://mlsanity.com/paper/2003.03581/) (2020)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)
- [Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis](https://mlsanity.com/paper/2003.11571/) (2020)
- [Creative Sketch Generation](https://mlsanity.com/paper/2011.10039/) (2020)
- [Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt](https://mlsanity.com/paper/2005.09159/) (2020)
- [Context-Aware Layout to Image Generation with Enhanced Object Appearance](https://mlsanity.com/paper/2103.11897/) (2021)
- [Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories](https://mlsanity.com/paper/1709.04121/) (2017)
- [SketchLattice: Latticed Representation for Sketch Manipulation](https://mlsanity.com/paper/2108.11636/) (2021)
- [Free-hand Sketch Synthesis with Deformable Stroke Models](https://mlsanity.com/paper/1510.02644/) (2015)
- [Teaching GANs to Sketch in Vector Format](https://mlsanity.com/paper/1904.03620/) (2019)
- [Learning to Sketch with Deep Q Networks and Demonstrated Strokes](https://mlsanity.com/paper/1810.05977/) (2018)
- [Rethinking conditional GAN training: An approach using geometrically structured latent manifolds](https://mlsanity.com/paper/2011.13055/) (2020)

## Built on by (most cited first, 17 of 17)

- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)
- [DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models](https://mlsanity.com/paper/2306.14685/) (2023)
- [Breathing Life Into Sketches Using Text-to-Video Priors](https://mlsanity.com/paper/2311.13608/) (2023)
- [SketchAgent: Language-Driven Sequential Sketch Generation](https://mlsanity.com/paper/2411.17673/) (2024)
- [Facial-Sketch Synthesis: A New Challenge](https://mlsanity.com/paper/2112.15439/) (2021)
- [SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition](https://mlsanity.com/paper/2405.03099/) (2024)
- [Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching](https://mlsanity.com/paper/2602.12280/) (2026)
- [VideoSketcher: Sequential Sketch Generation Using Video Model Priors](https://mlsanity.com/paper/2602.15819/) (2026)
- [Neural Collage Transfer: Artistic Reconstruction via Material Manipulation](https://mlsanity.com/paper/2311.02202/) (2023)
- [ShadowDraw: From Any Object to Shadow-Drawing Compositional Art](https://mlsanity.com/paper/2512.05110/) (2025)
- [CLIPDraw++: Text-to-Sketch Synthesis with Simple Primitives](https://mlsanity.com/paper/2312.02345/) (2023)
- [Hybrid Classical-Quantum architecture for vectorised image classification of hand-written sketches](https://mlsanity.com/paper/2407.06416/) (2024)
- [FedGAI: Federated Style Learning with Cloud-Edge Collaboration for Generative AI in Fashion Design](https://mlsanity.com/paper/2503.12389/) (2025)
- [Order Matters: 3D Shape Generation from Sequential VR Sketches](https://mlsanity.com/paper/2512.04761/) (2025)
- [SketchMamba: A Lightweight State-Space Model for Joint Progressive Sketch Classification and Stroke Auto-Completion](https://mlsanity.com/paper/2607.23580/) (2026)
- [Draw This First](https://mlsanity.com/paper/2608.12064/) (2026)
- [SketchFlow: Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space](https://mlsanity.com/paper/2608.21659/) (2026)

## Abstract (arXiv)

Creative sketching or doodling is an expressive activity, where imaginative and previously unseen depictions of everyday visual objects are drawn. Creative sketch image generation is a challenging vision problem, where the task is to generate diverse, yet realistic creative sketches possessing the unseen composition of the visual-world objects. Here, we propose a novel coarse-to-fine two-stage framework, DoodleFormer, that decomposes the creative sketch generation problem into the creation of coarse sketch composition followed by the incorporation of fine-details in the sketch. We introduce graph-aware transformer encoders that effectively capture global dynamic as well as local static structural relations among different body parts. To ensure diversity of the generated creative sketches, we introduce a probabilistic coarse sketch decoder that explicitly models the variations of each sketch body part to be drawn. Experiments are performed on two creative sketch datasets: Creative Birds and Creative Creatures. Our qualitative, quantitative and human-based evaluations show that DoodleFormer outperforms the state-of-the-art on both datasets, yielding realistic and diverse creative sketches. On Creative Creatures, DoodleFormer achieves an absolute gain of 25 in terms of Fr`echet inception distance (FID) over the state-of-the-art. We also demonstrate the effectiveness of DoodleFormer for related applications of text to creative sketch generation and sketch completion.