mlsanityData to 7 Oct 2026

Sketch-based Medical Image Retrieval

arXiv 2303.03633 ↗First posted 7 Mar 2023cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

The amount of medical images stored in hospitals is increasing faster than ever; however, utilizing the accumulated medical images has been limited. This is because existing content-based medical image retrieval (CBMIR) systems usually require example images to construct query vectors; nevertheless, example images cannot always be prepared. Besides, there can be images with rare characteristics that make it difficult to find similar example images, which we call isolated samples. Here, we introduce a novel sketch-based medical image retrieval (SBMIR) system that enables users to find images of interest without example images. The key idea lies in feature decomposition of medical images, whereby the entire feature of a medical image can be decomposed into and reconstructed from normal and abnormal features. By extending this idea, our SBMIR system provides an easy-to-use two-step graphical user interface: users first select a template image to specify a normal feature and then draw a semantic sketch of the disease on the template image to represent an abnormal feature. Subsequently, it integrates the two kinds of input to construct a query vector and retrieves reference images with the closest reference vectors. Using two datasets, ten healthcare professionals with various clinical backgrounds participated in the user test for evaluation. As a result, our SBMIR system enabled users to overcome previous challenges, including image retrieval based on fine-grained image characteristics, image retrieval without example images, and image retrieval for isolated samples. Our SBMIR system achieves flexible medical image retrieval on demand, thereby expanding the utility of medical image databases.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 19 of its 54 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

19 of 54 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  9. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  10. 2016The Importance of Skip Connections in Biomedical Image Segmentationcited by 126
  11. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  12. 2016SIFT Meets CNN: A Decade Survey of Instance Retrievalcited by 66
7 more
  1. 2019Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrievalcited by 57
  2. 2021Learning Disentangled Representations in the Imaging Domaincited by 46
  3. 2020Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factorscited by 39
  4. 2022Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrievalcited by 27
  5. 2021Iterative Human and Automated Identification of Wildlife Imagescited by 22
  6. 2020Deep Metric Learning-based Image Retrieval System for Chest Radiograph and its Clinical Applications in COVID-19cited by 12
  7. 2020Decomposing Normal and Abnormal Features of Medical Images for Content-based Image Retrievalcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 3.

6 papers.

  1. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  2. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 28
  3. 2023What Can Human Sketches Do for Object Detection?cited by 19
  4. 2025RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Miningcited by 4
  5. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  6. 2025SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketchescited by 1