# Multi-Graph Transformer for Free-Hand Sketch Recognition

arXiv: https://arxiv.org/abs/1912.11258 · HTML: https://mlsanity.com/paper/1912.11258/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2019.

## Built on by (most cited first, 30 of 35)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting](https://mlsanity.com/paper/2103.13716/) (2021)
- [How to Understand Masked Autoencoders](https://mlsanity.com/paper/2202.03670/) (2022)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [SketchLattice: Latticed Representation for Sketch Manipulation](https://mlsanity.com/paper/2108.11636/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Abstracting Sketches through Simple Primitives](https://mlsanity.com/paper/2207.13543/) (2022)
- [Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction](https://mlsanity.com/paper/2112.09385/) (2021)
- [Deep Learning in Concealed Dense Prediction](https://mlsanity.com/paper/2504.10979/) (2025)
- [Road Network Guided Fine-Grained Urban Traffic Flow Inference](https://mlsanity.com/paper/2109.14251/) (2021)
- [Edge Augmentation for Large-Scale Sketch Recognition without Sketches](https://mlsanity.com/paper/2202.13164/) (2022)
- [Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation](https://mlsanity.com/paper/2211.16841/) (2022)
- [VariGrad: A Novel Feature Vector Architecture for Geometric Deep Learning on Unregistered Data](https://mlsanity.com/paper/2307.03553/) (2023)
- [MeT: A Graph Transformer for Semantic Segmentation of 3D Meshes](https://mlsanity.com/paper/2307.01115/) (2023)
- [An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information](https://mlsanity.com/paper/2104.02605/) (2021)
- [Sketch Input Method Editor: A Comprehensive Dataset and Methodology for Systematic Input Recognition](https://mlsanity.com/paper/2311.18254/) (2023)
- [Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation](https://mlsanity.com/paper/2403.17525/) (2024)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [Class-Agnostic Visio-Temporal Scene Sketch Semantic Segmentation](https://mlsanity.com/paper/2410.00266/) (2024)
- [Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch](https://mlsanity.com/paper/2505.23763/) (2025)
- [InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping](https://mlsanity.com/paper/2106.06905/) (2021)
- [Font Completion and Manipulation by Cycling Between Multi-Modality Representations](https://mlsanity.com/paper/2108.12965/) (2021)
- [Leveraging Unlabeled Data for Sketch-based Understanding](https://mlsanity.com/paper/2204.12522/) (2022)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)
- [SigFormer: Sparse Signal-Guided Transformer for Multi-Modal Human Action Segmentation](https://mlsanity.com/paper/2311.17428/) (2023)
- [Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing](https://mlsanity.com/paper/2312.07875/) (2023)
- [Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts](https://mlsanity.com/paper/2607.07267/) (2026)
- [SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks](https://mlsanity.com/paper/2204.13153/) (2022)
- [TransFA: Transformer-based Representation for Face Attribute Evaluation](https://mlsanity.com/paper/2207.05456/) (2022)
- [Multimodal Query-guided Object Localization](https://mlsanity.com/paper/2212.00749/) (2022)

## Abstract (arXiv)

Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with Convolutional Neural Networks (CNNs) or the temporal sequential property with Recurrent Neural Networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel Graph Neural Network (GNN), the Multi-Graph Transformer (MGT), for learning representations of sketches from multiple graphs which simultaneously capture global and local geometric stroke structures, as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: (i) achieves small recognition gap to the CNN-based performance upper bound (72.80% vs. 74.22%), and (ii) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at https://github.com/PengBoXiangShang/multigraph_transformer.