# ViSketch-GPT: Collaborative Multi-Scale Feature Extraction for Sketch Recognition and Generation

arXiv: https://arxiv.org/abs/2503.22374 · HTML: https://mlsanity.com/paper/2503.22374/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [Creative Sketch Generation](https://mlsanity.com/paper/2011.10039/) (2020)
- [B'ezierSketch: A generative model for scalable vector sketches](https://mlsanity.com/paper/2007.02190/) (2020)
- [Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt](https://mlsanity.com/paper/2005.09159/) (2020)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval](https://mlsanity.com/paper/1804.01401/) (2018)
- [Multi-Graph Transformer for Free-Hand Sketch Recognition](https://mlsanity.com/paper/1912.11258/) (2019)
- [ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention](https://mlsanity.com/paper/2311.16682/) (2023)
- [Teaching GANs to Sketch in Vector Format](https://mlsanity.com/paper/1904.03620/) (2019)
- [Stroke-based sketched symbol reconstruction and segmentation](https://mlsanity.com/paper/1901.03427/) (2019)
- [Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers](https://mlsanity.com/paper/2308.14152/) (2023)
- [Sketch-a-Classifier: Sketch-based Photo Classifier Generation](https://mlsanity.com/paper/1804.11182/) (2018)
- [SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition](https://mlsanity.com/paper/2405.03099/) (2024)
- [Adversarial Training For Sketch Retrieval](https://mlsanity.com/paper/1607.02748/) (2016)

## Abstract (arXiv)

Understanding the nature of human sketches is challenging because of the wide variation in how they are created. Recognizing complex structural patterns improves both the accuracy in recognizing sketches and the fidelity of the generated sketches. In this work, we introduce ViSketch-GPT, a novel algorithm designed to address these challenges through a multi-scale context extraction approach. The model captures intricate details at multiple scales and combines them using an ensemble-like mechanism, where the extracted features work collaboratively to enhance the recognition and generation of key details crucial for classification and generation tasks. The effectiveness of ViSketch-GPT is validated through extensive experiments on the QuickDraw dataset. Our model establishes a new benchmark, significantly outperforming existing methods in both classification and generation tasks, with substantial improvements in accuracy and the fidelity of generated sketches. The proposed algorithm offers a robust framework for understanding complex structures by extracting features that collaborate to recognize intricate details, enhancing the understanding of structures like sketches and making it a versatile tool for various applications in computer vision and machine learning.