# SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval

arXiv: https://arxiv.org/abs/1804.01401 · HTML: https://mlsanity.com/paper/1804.01401/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2018.

## Built on by (most cited first, 30 of 35)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting](https://mlsanity.com/paper/2103.13716/) (2021)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style](https://mlsanity.com/paper/2303.14348/) (2023)
- [Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches](https://mlsanity.com/paper/2203.14843/) (2022)
- [FreestyleRet: Retrieving Images from Style-Diversified Queries](https://mlsanity.com/paper/2312.02428/) (2023)
- [What Can Human Sketches Do for Object Detection?](https://mlsanity.com/paper/2303.15149/) (2023)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [SketchINR: A First Look into Sketches as Implicit Neural Representations](https://mlsanity.com/paper/2403.09344/) (2024)
- [SketchXAI: A First Look at Explainability for Human Sketches](https://mlsanity.com/paper/2304.11744/) (2023)
- [Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers](https://mlsanity.com/paper/2403.07214/) (2024)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models](https://mlsanity.com/paper/2503.14129/) (2025)
- [Deep Asymmetric Hashing with Dual Semantic Regression and Class Structure Quantization](https://mlsanity.com/paper/2110.12478/) (2021)
- [Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval](https://mlsanity.com/paper/2111.10917/) (2021)
- [Automating Style Analysis and Visualization With Explainable AI -- Case Studies on Brand Recognition](https://mlsanity.com/paper/2306.03021/) (2023)
- [Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects](https://mlsanity.com/paper/2306.05832/) (2023)
- [Sketch Input Method Editor: A Comprehensive Dataset and Methodology for Systematic Input Recognition](https://mlsanity.com/paper/2311.18254/) (2023)
- [Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation](https://mlsanity.com/paper/2403.17525/) (2024)
- [U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models](https://mlsanity.com/paper/2403.18425/) (2024)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [Leveraging Unlabeled Data for Sketch-based Understanding](https://mlsanity.com/paper/2204.12522/) (2022)
- [Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing](https://mlsanity.com/paper/2312.07875/) (2023)
- [Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts](https://mlsanity.com/paper/2607.07267/) (2026)
- [On Training Sketch Recognizers for New Domains](https://mlsanity.com/paper/2104.08850/) (2021)
- [A hierarchical residual network with compact triplet-center loss for sketch recognition](https://mlsanity.com/paper/2109.13536/) (2021)
- [SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks](https://mlsanity.com/paper/2204.13153/) (2022)
- [Multimodal Query-guided Object Localization](https://mlsanity.com/paper/2212.00749/) (2022)

## Abstract (arXiv)

We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design.