# Abstracting Sketches through Simple Primitives

arXiv: https://arxiv.org/abs/2207.13543 · HTML: https://mlsanity.com/paper/2207.13543/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [Computer-Aided Design as Language](https://mlsanity.com/paper/2105.02769/) (2021)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [SketchGen: Generating Constrained CAD Sketches](https://mlsanity.com/paper/2106.02711/) (2021)
- [Vitruvion: A Generative Model of Parametric CAD Sketches](https://mlsanity.com/paper/2109.14124/) (2021)
- [Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2002.10310/) (2020)
- [More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2103.13990/) (2021)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [Multi-Graph Transformer for Free-Hand Sketch Recognition](https://mlsanity.com/paper/1912.11258/) (2019)
- [Goal-Driven Sequential Data Abstraction](https://mlsanity.com/paper/1907.12336/) (2019)
- [Physical Primitive Decomposition](https://mlsanity.com/paper/1809.05070/) (2018)
- [Asymmetric Feature Maps with Application to Sketch Based Retrieval](https://mlsanity.com/paper/1704.03946/) (2017)
- [Learning Manifold Patch-Based Representations of Man-Made Shapes](https://mlsanity.com/paper/1906.12337/) (2019)

## Built on by (most cited first, 14 of 14)

- [What Can Human Sketches Do for Object Detection?](https://mlsanity.com/paper/2303.15149/) (2023)
- [SketchINR: A First Look into Sketches as Implicit Neural Representations](https://mlsanity.com/paper/2403.09344/) (2024)
- [SketchXAI: A First Look at Explainability for Human Sketches](https://mlsanity.com/paper/2304.11744/) (2023)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [PICASSO: A Feed-Forward Framework for Parametric Inference of CAD Sketches via Rendering Self-Supervision](https://mlsanity.com/paper/2407.13394/) (2024)
- [Data-Free Sketch-Based Image Retrieval](https://mlsanity.com/paper/2303.07775/) (2023)
- [SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition](https://mlsanity.com/paper/2405.03099/) (2024)
- [Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching](https://mlsanity.com/paper/2602.12280/) (2026)
- [Historical Astronomical Diagrams Decomposition in Geometric Primitives](https://mlsanity.com/paper/2403.08721/) (2024)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [StrokeFusion: Vector Sketch Generation via Joint Stroke-UDF Encoding and Latent Sequence Diffusion](https://mlsanity.com/paper/2503.23752/) (2025)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)
- [CLIPDraw++: Text-to-Sketch Synthesis with Simple Primitives](https://mlsanity.com/paper/2312.02345/) (2023)
- [Hybrid Classical-Quantum architecture for vectorised image classification of hand-written sketches](https://mlsanity.com/paper/2407.06416/) (2024)

## Abstract (arXiv)

Humans show high-level of abstraction capabilities in games that require quickly communicating object information. They decompose the message content into multiple parts and communicate them in an interpretable protocol. Toward equipping machines with such capabilities, we propose the Primitive-based Sketch Abstraction task where the goal is to represent sketches using a fixed set of drawing primitives under the influence of a budget. To solve this task, our Primitive-Matching Network (PMN), learns interpretable abstractions of a sketch in a self supervised manner. Specifically, PMN maps each stroke of a sketch to its most similar primitive in a given set, predicting an affine transformation that aligns the selected primitive to the target stroke. We learn this stroke-to-primitive mapping end-to-end with a distance-transform loss that is minimal when the original sketch is precisely reconstructed with the predicted primitives. Our PMN abstraction empirically achieves the highest performance on sketch recognition and sketch-based image retrieval given a communication budget, while at the same time being highly interpretable. This opens up new possibilities for sketch analysis, such as comparing sketches by extracting the most relevant primitives that define an object category. Code is available at https://github.com/ExplainableML/sketch-primitives.