# LiveSketch: Query Perturbations for Guided Sketch-based Visual Search

arXiv: https://arxiv.org/abs/1904.06611 · HTML: https://mlsanity.com/paper/1904.06611/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2019.

## Built on by (most cited first, 30 of 34)

- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting](https://mlsanity.com/paper/2103.13716/) (2021)
- [More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2103.13990/) (2021)
- [Sketch3T: Test-Time Training for Zero-Shot SBIR](https://mlsanity.com/paper/2203.14691/) (2022)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch](https://mlsanity.com/paper/2208.03354/) (2022)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2104.05703/) (2021)
- [Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches](https://mlsanity.com/paper/2203.14843/) (2022)
- [Partially Does It: Towards Scene-Level FG-SBIR with Partial Input](https://mlsanity.com/paper/2203.14804/) (2022)
- [What Can Human Sketches Do for Object Detection?](https://mlsanity.com/paper/2303.15149/) (2023)
- [Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings](https://mlsanity.com/paper/2303.11502/) (2023)
- [Adaptive Fine-Grained Sketch-Based Image Retrieval](https://mlsanity.com/paper/2207.01723/) (2022)
- [Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR](https://mlsanity.com/paper/2303.13779/) (2023)
- [SketchINR: A First Look into Sketches as Implicit Neural Representations](https://mlsanity.com/paper/2403.09344/) (2024)
- [You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2403.07222/) (2024)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers](https://mlsanity.com/paper/2403.07214/) (2024)
- [Navigable Proximity Graph-Driven Native Hybrid Queries with Structured and Unstructured Constraints](https://mlsanity.com/paper/2203.13601/) (2022)
- [AniFaceDrawing: Anime Portrait Exploration during Your Sketching](https://mlsanity.com/paper/2306.07476/) (2023)
- [Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions](https://mlsanity.com/paper/2502.08438/) (2025)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models](https://mlsanity.com/paper/2503.14129/) (2025)
- [Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval](https://mlsanity.com/paper/2111.10917/) (2021)
- [Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval](https://mlsanity.com/paper/2401.04860/) (2024)
- [Do Generalised Classifiers really work on Human Drawn Sketches?](https://mlsanity.com/paper/2407.03893/) (2024)
- [Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch](https://mlsanity.com/paper/2505.23763/) (2025)
- [Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval](https://mlsanity.com/paper/2201.05007/) (2022)
- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)

## Abstract (arXiv)

LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch search by creating visual suggestions that augment the query as it is drawn, making query specification an iterative rather than one-shot process that helps disambiguate users' search intent. Our technical contributions are: a triplet convnet architecture that incorporates an RNN based variational autoencoder to search for images using vector (stroke-based) queries; real-time clustering to identify likely search intents (and so, targets within the search embedding); and the use of backpropagation from those targets to perturb the input stroke sequence, so suggesting alterations to the query in order to guide the search. We show improvements in accuracy and time-to-task over contemporary baselines using a 67M image corpus.