# Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval

arXiv: https://arxiv.org/abs/2201.05007 · HTML: https://mlsanity.com/paper/2201.05007/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 13 Jan 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches](https://mlsanity.com/paper/2003.03836/) (2020)
- [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1904.03451/) (2019)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval](https://mlsanity.com/paper/1703.05605/) (2017)
- [Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2002.10310/) (2020)
- [Learning to Sketch with Shortcut Cycle Consistency](https://mlsanity.com/paper/1805.00247/) (2018)
- [Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt](https://mlsanity.com/paper/2005.09159/) (2020)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [LiveSketch: Query Perturbations for Guided Sketch-based Visual Search](https://mlsanity.com/paper/1904.06611/) (2019)
- [Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2007.15103/) (2020)
- [CoSE: Compositional Stroke Embeddings](https://mlsanity.com/paper/2006.09930/) (2020)
- [Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories](https://mlsanity.com/paper/1709.04121/) (2017)
- [Asymmetric Feature Maps with Application to Sketch Based Retrieval](https://mlsanity.com/paper/1704.03946/) (2017)

## Built on by (most cited first, 1 of 1)

- [Towards Practicality of Sketch-Based Visual Understanding](https://mlsanity.com/paper/2210.15146/) (2022)

## Abstract (arXiv)

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo in a given query sketch. However, its widespread applicability is limited by the fact that it is difficult to draw a complete sketch for most people, and the drawing process often takes time. In this study, we aim to retrieve the target photo with the least number of strokes possible (incomplete sketch), named on-the-fly FG-SBIR (Bhunia et al. 2020), which starts retrieving at each stroke as soon as the drawing begins. We consider that there is a significant correlation among these incomplete sketches in the sketch drawing episode of each photo. To learn more efficient joint embedding space shared between the photo and its incomplete sketches, we propose a multi-granularity association learning framework that further optimizes the embedding space of all incomplete sketches. Specifically, based on the integrity of the sketch, we can divide a complete sketch episode into several stages, each of which corresponds to a simple linear mapping layer. Moreover, our framework guides the vector space representation of the current sketch to approximate that of its later sketches to realize the retrieval performance of the sketch with fewer strokes to approach that of the sketch with more strokes. In the experiments, we proposed more realistic challenges, and our method achieved superior early retrieval efficiency over the state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.