# Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval

arXiv: https://arxiv.org/abs/1903.03372 · HTML: https://mlsanity.com/paper/1903.03372/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2019.

## Built on by (most cited first, 30 of 57)

- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting](https://mlsanity.com/paper/2103.13716/) (2021)
- [More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2103.13990/) (2021)
- [Sketch3T: Test-Time Training for Zero-Shot SBIR](https://mlsanity.com/paper/2203.14691/) (2022)
- [A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch](https://mlsanity.com/paper/2208.03354/) (2022)
- [Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval](https://mlsanity.com/paper/2203.14817/) (2022)
- [Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style](https://mlsanity.com/paper/2303.14348/) (2023)
- [Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2104.05703/) (2021)
- [Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches](https://mlsanity.com/paper/2203.14843/) (2022)
- [Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings](https://mlsanity.com/paper/2303.11502/) (2023)
- [An Efficient Framework for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2102.04016/) (2021)
- [Adaptive Fine-Grained Sketch-Based Image Retrieval](https://mlsanity.com/paper/2207.01723/) (2022)
- [Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR](https://mlsanity.com/paper/2303.13779/) (2023)
- [Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval](https://mlsanity.com/paper/2106.11841/) (2021)
- [Operator SVD with Neural Networks via Nested Low-Rank Approximation](https://mlsanity.com/paper/2402.03655/) (2024)
- [You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2403.07222/) (2024)
- [Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval](https://mlsanity.com/paper/2202.11948/) (2022)
- [CrossATNet - A Novel Cross-Attention Based Framework for Sketch-Based Image Retrieval](https://mlsanity.com/paper/2104.09918/) (2021)
- [BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR](https://mlsanity.com/paper/2201.06570/) (2022)
- [Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers](https://mlsanity.com/paper/2403.07214/) (2024)
- [Zero-Shot Sketch Based Image Retrieval using Graph Transformer](https://mlsanity.com/paper/2201.10185/) (2022)
- [Data-Free Sketch-Based Image Retrieval](https://mlsanity.com/paper/2303.07775/) (2023)
- [Universal Cross-Domain Retrieval: Generalizing Across Classes and Domains](https://mlsanity.com/paper/2108.08356/) (2021)
- [Sketch-based Medical Image Retrieval](https://mlsanity.com/paper/2303.03633/) (2023)
- [Zero-Shot Multi-Modal Artist-Controlled Retrieval and Exploration of 3D Object Sets](https://mlsanity.com/paper/2209.00682/) (2022)
- [Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions](https://mlsanity.com/paper/2502.08438/) (2025)
- [Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval](https://mlsanity.com/paper/2210.10486/) (2022)
- [What Sketch Explainability Really Means for Downstream Tasks](https://mlsanity.com/paper/2403.09480/) (2024)
- [Composed Image Retrieval for Training-Free Domain Conversion](https://mlsanity.com/paper/2412.03297/) (2024)

## Abstract (arXiv)

Zero-shot sketch-based image retrieval (SBIR) is an emerging task in computer vision, allowing to retrieve natural images relevant to sketch queries that might not been seen in the training phase. Existing works either require aligned sketch-image pairs or inefficient memory fusion layer for mapping the visual information to a semantic space. In this work, we propose a semantically aligned paired cycle-consistent generative (SEM-PCYC) model for zero-shot SBIR, where each branch maps the visual information to a common semantic space via an adversarial training. Each of these branches maintains a cycle consistency that only requires supervision at category levels, and avoids the need of highly-priced aligned sketch-image pairs. A classification criteria on the generators' outputs ensures the visual to semantic space mapping to be discriminating. Furthermore, we propose to combine textual and hierarchical side information via a feature selection auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in zero-shot SBIR performance over the state-of-the-art on the challenging Sketchy and TU-Berlin datasets.