# Similar Image Search for Histopathology: SMILY

arXiv: https://arxiv.org/abs/1901.11112 · HTML: https://mlsanity.com/paper/1901.11112/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2019.

## Built on by (most cited first, 30 of 41)

- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [Towards a Visual-Language Foundation Model for Computational Pathology](https://mlsanity.com/paper/2307.12914/) (2023)
- [Evidence-based diagnostic reasoning with multi-agent copilot for human pathology](https://mlsanity.com/paper/2506.20964/) (2025)
- [A Foundational Multimodal Vision Language AI Assistant for Human Pathology](https://mlsanity.com/paper/2312.07814/) (2023)
- [On Image Search in Histopathology](https://mlsanity.com/paper/2401.08699/) (2024)
- [Self-Rule to Multi-Adapt: Generalized Multi-source Feature Learning Using Unsupervised Domain Adaptation for Colorectal Cancer Tissue Detection](https://mlsanity.com/paper/2108.09178/) (2021)
- [Learning Binary and Sparse Permutation-Invariant Representations for Fast and Memory Efficient Whole Slide Image Search](https://mlsanity.com/paper/2208.13653/) (2022)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [Analysis and Validation of Image Search Engines in Histopathology](https://mlsanity.com/paper/2401.03271/) (2024)
- [Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review](https://mlsanity.com/paper/2112.12705/) (2021)
- [Interpretable Prediction of Lung Squamous Cell Carcinoma Recurrence With Self-supervised Learning](https://mlsanity.com/paper/2203.12204/) (2022)
- [Evolutionary Computation in Action: Feature Selection for Deep Embedding Spaces of Gigapixel Pathology Images](https://mlsanity.com/paper/2303.00943/) (2023)
- [Towards More Transparent and Accurate Cancer Diagnosis with an Unsupervised CAE Approach](https://mlsanity.com/paper/2305.11728/) (2023)
- [SPLICE -- Streamlining Digital Pathology Image Processing](https://mlsanity.com/paper/2404.17704/) (2024)
- [Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval](https://mlsanity.com/paper/2409.09430/) (2024)
- [Case-based Similar Image Retrieval for Weakly Annotated Large Histopathological Images of Malignant Lymphoma Using Deep Metric Learning](https://mlsanity.com/paper/2107.03602/) (2021)
- [Cross-Modality Sub-Image Retrieval using Contrastive Multimodal Image Representations](https://mlsanity.com/paper/2201.03597/) (2022)
- [Ranking Loss and Sequestering Learning for Reducing Image Search Bias in Histopathology](https://mlsanity.com/paper/2304.08498/) (2023)
- [WWFedCBMIR: World-Wide Federated Content-Based Medical Image Retrieval](https://mlsanity.com/paper/2305.03383/) (2023)
- [Pan-cancer Histopathology WSI Pre-training with Position-aware Masked Autoencoder](https://mlsanity.com/paper/2407.07504/) (2024)
- [Transformer-based Personalized Attention Mechanism for Medical Images with Clinical Records](https://mlsanity.com/paper/2206.03003/) (2022)
- [Selection of Distinct Morphologies to Divide & Conquer Gigapixel Pathology Images](https://mlsanity.com/paper/2311.09902/) (2023)
- [Siamese Content-based Search Engine for a More Transparent Skin and Breast Cancer Diagnosis through Histological Imaging](https://mlsanity.com/paper/2401.08272/) (2024)
- [Leveraging Medical Foundation Model Features in Graph Neural Network-Based Retrieval of Breast Histopathology Images](https://mlsanity.com/paper/2405.04211/) (2024)
- [An Interpretable Algorithm for Uveal Melanoma Subtyping from Whole Slide Cytology Images](https://mlsanity.com/paper/2108.06246/) (2021)
- [Immunohistochemistry Biomarkers-Guided Image Search for Histopathology](https://mlsanity.com/paper/2304.12424/) (2023)
- [A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models](https://mlsanity.com/paper/2308.01328/) (2023)

## Abstract (arXiv)

The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Though these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. Because pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep learning based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY's ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist's arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.