# Differentiable Drawing and Sketching

arXiv: https://arxiv.org/abs/2103.16194 · HTML: https://mlsanity.com/paper/2103.16194/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [Neural Painters: A learned differentiable constraint for generating brushstroke paintings](https://mlsanity.com/paper/1904.08410/) (2019)
- [Unsupervised Doodling and Painting with Improved SPIRAL](https://mlsanity.com/paper/1910.01007/) (2019)
- [Universal Perceptual Grouping](https://mlsanity.com/paper/1808.02312/) (2018)
- [Teaching GANs to Sketch in Vector Format](https://mlsanity.com/paper/1904.03620/) (2019)
- [SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks](https://mlsanity.com/paper/2003.00678/) (2020)
- [Adversarial Training For Sketch Retrieval](https://mlsanity.com/paper/1607.02748/) (2016)
- [SketchParse : Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks](https://mlsanity.com/paper/1709.01295/) (2017)

## Built on by (most cited first, 21 of 21)

- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [Learning to Draw: Emergent Communication through Sketching](https://mlsanity.com/paper/2106.02067/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [NeuralSVG: An Implicit Representation for Text-to-Vector Generation](https://mlsanity.com/paper/2501.03992/) (2025)
- [SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation](https://mlsanity.com/paper/2308.14191/) (2023)
- [Few-shot Geometry-Aware Keypoint Localization](https://mlsanity.com/paper/2303.17216/) (2023)
- [Learning Geometry-aware Representations by Sketching](https://mlsanity.com/paper/2304.08204/) (2023)
- [Stroke-based Rendering: From Heuristics to Deep Learning](https://mlsanity.com/paper/2302.00595/) (2022)
- [Neural Image Abstraction Using Long Smoothing B-Splines](https://mlsanity.com/paper/2511.05360/) (2025)
- [Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimation](https://mlsanity.com/paper/2312.07051/) (2023)
- [Geometric Features Enhanced Human-Object Interaction Detection](https://mlsanity.com/paper/2406.18691/) (2024)
- [Contrastive Multimodal Learning for Emergence of Graphical Sensory-Motor Communication](https://mlsanity.com/paper/2210.06468/) (2022)
- [Human-Producible Adversarial Examples](https://mlsanity.com/paper/2310.00438/) (2023)
- [Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?](https://mlsanity.com/paper/2110.08203/) (2021)
- [Physically Embodied Deep Image Optimisation](https://mlsanity.com/paper/2201.08142/) (2022)
- [Differentiable Iterated Function Systems](https://mlsanity.com/paper/2203.01231/) (2022)
- [Optimize and Reduce: A Top-Down Approach for Image Vectorization](https://mlsanity.com/paper/2312.11334/) (2023)
- [Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillation](https://mlsanity.com/paper/2410.14700/) (2024)
- [Generative Visual Communication in the Era of Vision-Language Models](https://mlsanity.com/paper/2411.18727/) (2024)
- [Differentiable Stroke Planning with Dual Parameterization for Efficient and High-Fidelity Painting Creation](https://mlsanity.com/paper/2604.02752/) (2026)
- [Drawing with Strangers: Population Scaling Drives Zero-Shot Mutual Intelligibility in Emergent Sketching](https://mlsanity.com/paper/2606.10582/) (2026)

## Abstract (arXiv)

We present a bottom-up differentiable relaxation of the process of drawing points, lines and curves into a pixel raster. Our approach arises from the observation that rasterising a pixel in an image given parameters of a primitive can be reformulated in terms of the primitive's distance transform, and then relaxed to allow the primitive's parameters to be learned. This relaxation allows end-to-end differentiable programs and deep networks to be learned and optimised and provides several building blocks that allow control over how a compositional drawing process is modelled. We emphasise the bottom-up nature of our proposed approach, which allows for drawing operations to be composed in ways that can mimic the physical reality of drawing rather than being tied to, for example, approaches in modern computer graphics. With the proposed approach we demonstrate how sketches can be generated by directly optimising against photographs and how auto-encoders can be built to transform rasterised handwritten digits into vectors without supervision. Extensive experimental results highlight the power of this approach under different modelling assumptions for drawing tasks.