# CrossATNet - A Novel Cross-Attention Based Framework for Sketch-Based Image Retrieval

arXiv: https://arxiv.org/abs/2104.09918 · HTML: https://mlsanity.com/paper/2104.09918/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [Generalized Zero-Shot Learning via Synthesized Examples](https://mlsanity.com/paper/1712.03878/) (2017)
- [Zero-Shot Learning via Semantic Similarity Embedding](https://mlsanity.com/paper/1509.04767/) (2015)
- [A Zero-Shot Framework for Sketch-based Image Retrieval](https://mlsanity.com/paper/1807.11724/) (2018)
- [Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval](https://mlsanity.com/paper/1703.05605/) (2017)
- [Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1903.03372/) (2019)
- [Zero-Shot Sketch-Image Hashing](https://mlsanity.com/paper/1803.02284/) (2018)
- [Sketch-based 3D Shape Retrieval using Convolutional Neural Networks](https://mlsanity.com/paper/1504.03504/) (2015)
- [Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/1904.03208/) (2019)
- [Zero-Shot Learning via Joint Latent Similarity Embedding](https://mlsanity.com/paper/1511.04512/) (2015)
- [Recent Advances in Zero-shot Recognition](https://mlsanity.com/paper/1710.04837/) (2017)
- [Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos](https://mlsanity.com/paper/1910.08732/) (2019)
- [Stacked Adversarial Network for Zero-Shot Sketch based Image Retrieval](https://mlsanity.com/paper/2001.06657/) (2020)
- [CMIR-NET : A Deep Learning Based Model For Cross-Modal Retrieval In Remote Sensing](https://mlsanity.com/paper/1904.04794/) (2019)

## Built on by (most cited first, 9 of 9)

- [BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR](https://mlsanity.com/paper/2201.06570/) (2022)
- [Zero-Shot Sketch Based Image Retrieval using Graph Transformer](https://mlsanity.com/paper/2201.10185/) (2022)
- [Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition](https://mlsanity.com/paper/2202.07901/) (2022)
- [Learning Efficient Representations for Image-Based Patent Retrieval](https://mlsanity.com/paper/2308.13749/) (2023)
- [ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2111.12757/) (2021)
- [Transformers and CNNs both Beat Humans on SBIR](https://mlsanity.com/paper/2209.06629/) (2022)
- [If At First You Don't Succeed: Test Time Re-ranking for Zero-shot, Cross-domain Retrieval](https://mlsanity.com/paper/2303.17703/) (2023)
- [Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval](https://mlsanity.com/paper/2401.04860/) (2024)
- [A Recipe for Efficient SBIR Models: Combining Relative Triplet Loss with Batch Normalization and Knowledge Distillation](https://mlsanity.com/paper/2305.18988/) (2023)

## Abstract (arXiv)

We propose a novel framework for cross-modal zero-shot learning (ZSL) in the context of sketch-based image retrieval (SBIR). Conventionally, the SBIR schema mainly considers simultaneous mappings among the two image views and the semantic side information. Therefore, it is desirable to consider fine-grained classes mainly in the sketch domain using highly discriminative and semantically rich feature space. However, the existing deep generative modeling-based SBIR approaches majorly focus on bridging the gaps between the seen and unseen classes by generating pseudo-unseen-class samples. Besides, violating the ZSL protocol by not utilizing any unseen-class information during training, such techniques do not pay explicit attention to modeling the discriminative nature of the shared space. Also, we note that learning a unified feature space for both the multi-view visual data is a tedious task considering the significant domain difference between sketches and color images. In this respect, as a remedy, we introduce a novel framework for zero-shot SBIR. While we define a cross-modal triplet loss to ensure the discriminative nature of the shared space, an innovative cross-modal attention learning strategy is also proposed to guide feature extraction from the image domain exploiting information from the respective sketch counterpart. In order to preserve the semantic consistency of the shared space, we consider a graph CNN-based module that propagates the semantic class topology to the shared space. To ensure an improved response time during inference, we further explore the possibility of representing the shared space in terms of hash codes. Experimental results obtained on the benchmark TU-Berlin and the Sketchy datasets confirm the superiority of CrossATNet in yielding state-of-the-art results.