mlsanityData to 7 Oct 2026

Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style

arXiv 2303.14348 ↗First posted 25 Mar 2023cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

This paper studies the problem of zero-short sketch-based image retrieval (ZS-SBIR), however with two significant differentiators to prior art (i) we tackle all variants (inter-category, intra-category, and cross datasets) of ZS-SBIR with just one network (``everything''), and (ii) we would really like to understand how this sketch-photo matching operates (``explainable''). Our key innovation lies with the realization that such a cross-modal matching problem could be reduced to comparisons of groups of key local patches -- akin to the seasoned ``bag-of-words'' paradigm. Just with this change, we are able to achieve both of the aforementioned goals, with the added benefit of no longer requiring external semantic knowledge. Technically, ours is a transformer-based cross-modal network, with three novel components (i) a self-attention module with a learnable tokenizer to produce visual tokens that correspond to the most informative local regions, (ii) a cross-attention module to compute local correspondences between the visual tokens across two modalities, and finally (iii) a kernel-based relation network to assemble local putative matches and produce an overall similarity metric for a sketch-photo pair. Experiments show ours indeed delivers superior performances across all ZS-SBIR settings. The all important explainable goal is elegantly achieved by visualizing cross-modal token correspondences, and for the first time, via sketch to photo synthesis by universal replacement of all matched photo patches. Code and model are available at urlhttps://github.com/buptLinfy/ZSE-SBIR.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 43 of its 68 references that are in the map. First posted on arXiv 25 Mar 2023.

Builds on

43 of 68 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  7. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  8. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  9. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  10. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  11. 2016LIFT: Learned Invariant Feature Transformcited by 305
  12. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
31 more
  1. 2018LF-Net: Learning Local Features from Imagescited by 151
  2. 2017Semantic Autoencoder for Zero-Shot Learningcited by 133
  3. 2020Cross-domain Correspondence Learning for Exemplar-based Image Translationcited by 116
  4. 2016Synthesized Classifiers for Zero-Shot Learningcited by 113
  5. 2016Universal Correspondence Networkcited by 82
  6. 2019Doodle to Search: Practical Zero-Shot Sketch-based Image Retrievalcited by 79
  7. 2015Zero-Shot Learning via Semantic Similarity Embeddingcited by 74
  8. 2018A Zero-Shot Framework for Sketch-based Image Retrievalcited by 70
  9. 2017Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrievalcited by 68
  10. 2019SFNet: Learning Object-aware Semantic Correspondencecited by 60
  11. 2020Deep Learning for Free-Hand Sketch: A Surveycited by 59
  12. 2020Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrievalcited by 59
  13. 2015Proposal Flowcited by 57
  14. 2019Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrievalcited by 57
  15. 2021StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrievalcited by 50
  16. 2017SCNet: Learning Semantic Correspondencecited by 48
  17. 2018Zero-Shot Sketch-Image Hashingcited by 43
  18. 2017FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondencecited by 42
  19. 2019Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrievalcited by 40
  20. 2018SketchMate: Deep Hashing for Million-Scale Human Sketch Retrievalcited by 35
  21. 2022Sketch3T: Test-Time Training for Zero-Shot SBIRcited by 34
  22. 2014Do Convnets Learn Correspondence?cited by 28
  23. 2016Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classificationcited by 22
  24. 2021The Animation Transformer: Visual Correspondence via Segment Matchingcited by 20
  25. 2022Marginal Contrastive Correspondence for Guided Image Generationcited by 19
  26. 2016Zero-Shot Hashing via Transferring Supervised Knowledgecited by 15
  27. 2021An Efficient Framework for Zero-Shot Sketch-Based Image Retrievalcited by 15
  28. 2021Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrievalcited by 13
  29. 2022BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIRcited by 8
  30. 2015Image Reconstruction from Bag-of-Visual-Wordscited by 6
  31. 2018Neural Best-Buddies: Sparse Cross-Domain Correspondencecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 5; 2026: 1.

25 papers.

Sort:

By area:

  1. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  2. 2024Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakerscited by 8
  3. 2023Unveiling Objects with SOLA: An Annotation-Free Image Search on the Object Level for Automotive Data Setscited by 5
  4. 2024What Sketch Explainability Really Means for Downstream Taskscited by 4
  5. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  6. 2023Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrievalcited by 3
  7. 2023Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrievalcited by 3
  8. 2024Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learningcited by 3
  9. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  10. 2024Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrievalcited by 2
  11. 2024Relational Representation Learning Network for Cross-Spectral Image Patch Matchingcited by 2
  12. 2024CriSp: Leveraging Tread Depth Maps for Enhanced Crime-Scene Shoeprint Matchingcited by 2
13 more
  1. 2024Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrievalcited by 2
  2. 2024AirSketch: Generative Motion to Sketchcited by 2
  3. 2024Why and How: Knowledge-Guided Learning for Cross-Spectral Image Patch Matchingcited by 2
  4. 2024SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generationcited by 1
  5. 2024Relation-Aware Meta-Learning for Zero-shot Sketch-Based Image Retrievalcited by 1
  6. 2025FIGROTD: A Friendly-to-Handle Dataset for Image Guided Retrieval with Optional Textcited by 1
  7. 2023Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrievalcited by 0
  8. 2024Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenizationcited by 0
  9. 2024Dual-Modal Prompting for Sketch-Based Image Retrievalcited by 0
  10. 2025CLAIR: CLIP-Aided Weakly Supervised Zero-Shot Cross-Domain Image Retrievalcited by 0
  11. 2025Dynamic Multi-level Weighted Alignment Network for Zero-shot Sketch-based Image Retrievalcited by 0
  12. 2025UNION: A Lightweight Target Representation for Efficient Zero-Shot Image-Guided Retrieval with Optional Textual Queriescited by 0
  13. 2026SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrievalcited by 0