# Revisiting Medical Image Retrieval via Knowledge Consolidation

arXiv: https://arxiv.org/abs/2503.09370 · HTML: https://mlsanity.com/paper/2503.09370/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 16 of 16)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [Lizard: A Large-Scale Dataset for Colonic Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2108.11195/) (2021)
- [Central Similarity Quantization for Efficient Image and Video Retrieval](https://mlsanity.com/paper/1908.00347/) (2019)
- [RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods](https://mlsanity.com/paper/2301.05768/) (2023)
- [Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need](https://mlsanity.com/paper/2302.02615/) (2023)
- [The University of California San Francisco Preoperative Diffuse Glioma MRI (UCSF-PDGM) Dataset](https://mlsanity.com/paper/2109.00356/) (2021)
- [Frozen Transformers in Language Models Are Effective Visual Encoder Layers](https://mlsanity.com/paper/2310.12973/) (2023)
- [Asymmetric Deep Supervised Hashing](https://mlsanity.com/paper/1707.08325/) (2017)
- [Learning Super-Features for Image Retrieval](https://mlsanity.com/paper/2201.13182/) (2022)
- [Deep Multi-View Enhancement Hashing for Image Retrieval](https://mlsanity.com/paper/2002.00169/) (2020)
- [Generalized Product Quantization Network for Semi-supervised Image Retrieval](https://mlsanity.com/paper/2002.11281/) (2020)
- [Data Harmonisation for Information Fusion in Digital Healthcare: A State-of-the-Art Systematic Review, Meta-Analysis and Future Research Directions](https://mlsanity.com/paper/2201.06505/) (2022)
- [Deep Triplet Hashing Network for Case-based Medical Image Retrieval](https://mlsanity.com/paper/2101.12346/) (2021)
- [Unsupervised Tissue Segmentation via Deep Constrained Gaussian Network](https://mlsanity.com/paper/2208.02912/) (2022)

## Built on by (most cited first, 3 of 3)

- [HMAR: Hierarchical Modality-Aware Expert and Dynamic Routing Medical Image Retrieval Architecture](https://mlsanity.com/paper/2603.16679/) (2026)
- [MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education](https://mlsanity.com/paper/2605.04772/) (2026)
- [Weakly Supervised Pathology-Informed Representation Learning for PET-Based Content Retrieval of Intra-Tumour Heterogeneity](https://mlsanity.com/paper/2607.18762/) (2026)

## Abstract (arXiv)

As artificial intelligence and digital medicine increasingly permeate healthcare systems, robust governance frameworks are essential to ensure ethical, secure, and effective implementation. In this context, medical image retrieval becomes a critical component of clinical data management, playing a vital role in decision-making and safeguarding patient information. Existing methods usually learn hash functions using bottleneck features, which fail to produce representative hash codes from blended embeddings. Although contrastive hashing has shown superior performance, current approaches often treat image retrieval as a classification task, using category labels to create positive/negative pairs. Moreover, many methods fail to address the out-of-distribution (OOD) issue when models encounter external OOD queries or adversarial attacks. In this work, we propose a novel method to consolidate knowledge of hierarchical features and optimisation functions. We formulate the knowledge consolidation by introducing Depth-aware Representation Fusion (DaRF) and Structure-aware Contrastive Hashing (SCH). DaRF adaptively integrates shallow and deep representations into blended features, and SCH incorporates image fingerprints to enhance the adaptability of positive/negative pairings. These blended features further facilitate OOD detection and content-based recommendation, contributing to a secure AI-driven healthcare environment. Moreover, we present a content-guided ranking to improve the robustness and reproducibility of retrieval results. Our comprehensive assessments demonstrate that the proposed method could effectively recognise OOD samples and significantly outperform existing approaches in medical image retrieval (p<0.05). In particular, our method achieves a 5.6-38.9% improvement in mean Average Precision on the anatomical radiology dataset.