# Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval

arXiv: https://arxiv.org/abs/2204.05666 · HTML: https://mlsanity.com/paper/2204.05666/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1904.03451/) (2019)
- [Bi-Directional Cascade Network for Perceptual Edge Detection](https://mlsanity.com/paper/1902.10903/) (2019)
- [A Zero-Shot Framework for Sketch-based Image Retrieval](https://mlsanity.com/paper/1807.11724/) (2018)
- [Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2003.13261/) (2020)
- [Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval](https://mlsanity.com/paper/1703.05605/) (2017)
- [Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks](https://mlsanity.com/paper/1712.01928/) (2017)
- [Fast Edge Detection Using Structured Forests](https://mlsanity.com/paper/1406.5549/) (2014)
- [Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2002.10310/) (2020)
- [Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1903.03372/) (2019)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [Zero-Shot Sketch-Image Hashing](https://mlsanity.com/paper/1803.02284/) (2018)
- [Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/1904.03208/) (2019)
- [Discriminative Learning of Latent Features for Zero-Shot Recognition](https://mlsanity.com/paper/1803.06731/) (2018)
- [From Zero-shot Learning to Conventional Supervised Classification: Unseen Visual Data Synthesis](https://mlsanity.com/paper/1705.01782/) (2017)
- [Zero-Shot Hashing via Transferring Supervised Knowledge](https://mlsanity.com/paper/1606.05032/) (2016)
- [An Efficient Framework for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2102.04016/) (2021)
- [Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2106.11841/) (2021)
- [Cross-Paced Representation Learning with Partial Curricula for Sketch-based Image Retrieval](https://mlsanity.com/paper/1803.01504/) (2018)

## Built on by (most cited first, 2 of 2)

- [Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2305.05144/) (2023)
- [If At First You Don't Succeed: Test Time Re-ranking for Zero-shot, Cross-domain Retrieval](https://mlsanity.com/paper/2303.17703/) (2023)

## Abstract (arXiv)

The Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) is a challenging task because of the large domain gap between sketches and natural images as well as the semantic inconsistency between seen and unseen categories. Previous literature bridges seen and unseen categories by semantic embedding, which requires prior knowledge of the exact class names and additional extraction efforts. And most works reduce domain gap by mapping sketches and natural images into a common high-level space using constructed sketch-image pairs, which ignore the unpaired information between images and sketches. To address these issues, in this paper, we propose a novel Three-Stream Joint Training Network (3JOIN) for the ZS-SBIR task. To narrow the domain differences between sketches and images, we extract edge maps for natural images and treat them as a bridge between images and sketches, which have similar content to images and similar style to sketches. For exploiting a sufficient combination of sketches, natural images, and edge maps, a novel three-stream joint training network is proposed. In addition, we use a teacher network to extract the implicit semantics of the samples without the aid of other semantics and transfer the learned knowledge to unseen classes. Extensive experiments conducted on two real-world datasets demonstrate the superiority of our proposed method.