# Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories

arXiv: https://arxiv.org/abs/1709.04121 · HTML: https://mlsanity.com/paper/1709.04121/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2017.

## Built on by (most cited first, 20 of 20)

- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [Breathing Life Into Sketches Using Text-to-Video Priors](https://mlsanity.com/paper/2311.13608/) (2023)
- [More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2103.13990/) (2021)
- [SketchLattice: Latticed Representation for Sketch Manipulation](https://mlsanity.com/paper/2108.11636/) (2021)
- [DoodleFormer: Creative Sketch Drawing with Transformers](https://mlsanity.com/paper/2112.03258/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation](https://mlsanity.com/paper/2502.08642/) (2025)
- [SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation](https://mlsanity.com/paper/2308.14191/) (2023)
- [Multi-Object Sketch Animation with Grouping and Motion Trajectory Priors](https://mlsanity.com/paper/2508.15535/) (2025)
- [Learning Geometry-aware Representations by Sketching](https://mlsanity.com/paper/2304.08204/) (2023)
- [Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation](https://mlsanity.com/paper/2211.16841/) (2022)
- [Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation](https://mlsanity.com/paper/2403.17525/) (2024)
- [AirSketch: Generative Motion to Sketch](https://mlsanity.com/paper/2407.08906/) (2024)
- [Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval](https://mlsanity.com/paper/2201.05007/) (2022)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)
- [Character Generation through Self-Supervised Vectorization](https://mlsanity.com/paper/2208.02012/) (2022)
- [Generative Visual Communication in the Era of Vision-Language Models](https://mlsanity.com/paper/2411.18727/) (2024)
- [Text to Sketch Generation with Multi-Styles](https://mlsanity.com/paper/2511.04123/) (2025)
- [SketchXplain: Intuitive Visual Explanations of Image Classifiers with Sketches](https://mlsanity.com/paper/2606.17646/) (2026)
- [SketchMamba: A Lightweight State-Space Model for Joint Progressive Sketch Classification and Stroke Auto-Completion](https://mlsanity.com/paper/2607.23580/) (2026)

## Abstract (arXiv)

Sketch is an important media for human to communicate ideas, which reflects the superiority of human intelligence. Studies on sketch can be roughly summarized into recognition and generation. Existing models on image recognition failed to obtain satisfying performance on sketch classification. But for sketch generation, a recent study proposed a sequence-to-sequence variational-auto-encoder (VAE) model called sketch-rnn which was able to generate sketches based on human inputs. The model achieved amazing results when asked to learn one category of object, such as an animal or a vehicle. However, the performance dropped when multiple categories were fed into the model. Here, we proposed a model called sketch-pix2seq which could learn and draw multiple categories of sketches. Two modifications were made to improve the sketch-rnn model: one is to replace the bidirectional recurrent neural network (BRNN) encoder with a convolutional neural network(CNN); the other is to remove the Kullback-Leibler divergence from the objective function of VAE. Experimental results showed that models with CNN encoders outperformed those with RNN encoders in generating human-style sketches. Visualization of the latent space illustrated that the removal of KL-divergence made the encoder learn a posterior of latent space that reflected the features of different categories. Moreover, the combination of CNN encoder and removal of KL-divergence, i.e., the sketch-pix2seq model, had better performance in learning and generating sketches of multiple categories and showed promising results in creativity tasks.