mlsanityData to 7 Oct 2026

Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR

arXiv 2303.13779 ↗First posted 24 Mar 2023cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

This paper advances the fine-grained sketch-based image retrieval (FG-SBIR) literature by putting forward a strong baseline that overshoots prior state-of-the-arts by ~11%. This is not via complicated design though, but by addressing two critical issues facing the community (i) the gold standard triplet loss does not enforce holistic latent space geometry, and (ii) there are never enough sketches to train a high accuracy model. For the former, we propose a simple modification to the standard triplet loss, that explicitly enforces separation amongst photos/sketch instances. For the latter, we put forward a novel knowledge distillation module can leverage photo data for model training. Both modules are then plugged into a novel plug-n-playable training paradigm that allows for more stable training. More specifically, for (i) we employ an intra-modal triplet loss amongst sketches to bring sketches of the same instance closer from others, and one more amongst photos to push away different photo instances while bringing closer a structurally augmented version of the same photo (offering a gain of ~4-6%). To tackle (ii), we first pre-train a teacher on the large set of unlabelled photos over the aforementioned intra-modal photo triplet loss. Then we distill the contextual similarity present amongst the instances in the teacher's embedding space to that in the student's embedding space, by matching the distribution over inter-feature distances of respective samples in both embedding spaces (delivering a further gain of ~4-5%). Apart from outperforming prior arts significantly, our model also yields satisfactory results on generalising to new classes. Project page: https://aneeshan95.github.io/Sketch_PVT/

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 64 of its 93 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

64 of 93 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  6. 2016Colorful Image Colorizationcited by 1,113
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2019When Does Label Smoothing Help?cited by 985
  10. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  11. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  12. 2019Relational Knowledge Distillationcited by 873
  13. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  14. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  15. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  16. 2013Do Deep Nets Really Need to be Deep?cited by 466
  17. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  18. 2017A Neural Representation of Sketch Drawingscited by 311
  19. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  20. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  21. 2019An Attentive Survey of Attention Modelscited by 123
  22. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  23. 2022Learning to generate line drawings that convey geometry and semanticscited by 82
  24. 2019Doodle to Search: Practical Zero-Shot Sketch-based Image Retrievalcited by 79
  25. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  26. 2020Sketchformer: Transformer-based Representation for Sketched Structurecited by 78
  27. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78
  28. 2019Relation Distillation Networks for Video Object Detectioncited by 77
  29. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  30. 2018A Zero-Shot Framework for Sketch-based Image Retrievalcited by 70
  31. 2017Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrievalcited by 68
  32. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  33. 2020Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrievalcited by 59
  34. 2019Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrievalcited by 57
  35. 2015On Sparse variational methods and the Kullback-Leibler divergence between stochastic processescited by 56
  36. 2019Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorizationcited by 51
  37. 2021StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrievalcited by 50
  38. 2021Recall@k Surrogate Loss with Large Batches and Similarity Mixupcited by 49

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 8; 2025: 3.

14 papers.

Sort:

By area:

  1. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 28
  2. 2023What Can Human Sketches Do for Object Detection?cited by 19
  3. 202415M Multimodal Facial Image-Text Datasetcited by 14
  4. 2024You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrievalcited by 12
  5. 2024How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?cited by 10
  6. 2024Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakerscited by 8
  7. 2023ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrievalcited by 6
  8. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  9. 2024Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learningcited by 3
  10. 2023Active Learning for Fine-Grained Sketch-Based Image Retrievalcited by 2
  11. 2024Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrievalcited by 2
  12. 2024Freeview Sketching: View-Aware Fine-Grained Sketch-Based Image Retrievalcited by 2
2 more
  1. 2025Sketch Down the FLOPs: Towards Efficient Networks for Human Sketchcited by 2
  2. 2025Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detectioncited by 0