# Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query

arXiv: https://arxiv.org/abs/2305.05598 · HTML: https://mlsanity.com/paper/2305.05598/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 12 of 12)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation](https://mlsanity.com/paper/2102.10663/) (2021)
- [Medical Image Retrieval using Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.08472/) (2017)
- [Dataset Growth in Medical Image Analysis Research](https://mlsanity.com/paper/1908.07765/) (2019)
- [Deep Metric Learning-based Image Retrieval System for Chest Radiograph and its Clinical Applications in COVID-19](https://mlsanity.com/paper/2012.03663/) (2020)
- [Semantic-Aware Contrastive Learning for Multi-object Medical Image Segmentation](https://mlsanity.com/paper/2106.01596/) (2021)
- [Adaptive Contrastive Learning with Dynamic Correlation for Multi-Phase Organ Segmentation](https://mlsanity.com/paper/2210.08652/) (2022)

## Built on by (most cited first, 6 of 6)

- [3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology](https://mlsanity.com/paper/2311.13752/) (2023)
- [Leveraging Foundation Models for Content-Based Image Retrieval in Radiology](https://mlsanity.com/paper/2403.06567/) (2024)
- [RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining](https://mlsanity.com/paper/2503.04653/) (2025)
- [M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision](https://mlsanity.com/paper/2509.01360/) (2025)
- [WristMIR: Coarse-to-Fine Region-Aware Retrieval of Pediatric Wrist Radiographs with Radiology Report-Driven Learning](https://mlsanity.com/paper/2602.07872/) (2026)
- [RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models](https://mlsanity.com/paper/2609.16847/) (2026)

## Abstract (arXiv)

We introduce a novel Region-based contrastive pretraining for Medical Image Retrieval (RegionMIR) that demonstrates the feasibility of medical image retrieval with similar anatomical regions. RegionMIR addresses two major challenges for medical image retrieval i) standardization of clinically relevant searching criteria (e.g., anatomical, pathology-based), and ii) localization of anatomical area of interests that are semantically meaningful. In this work, we propose an ROI image retrieval image network that retrieves images with similar anatomy by extracting anatomical features (via bounding boxes) and evaluate similarity between pairwise anatomy-categorized features between the query and the database of images using contrastive learning. ROI queries are encoded using a contrastive-pretrained encoder that was fine-tuned for anatomy classification, which generates an anatomical-specific latent space for region-correlated image retrieval. During retrieval, we compare the anatomically encoded query to find similar features within a feature database generated from training samples, and retrieve images with similar regions from training samples. We evaluate our approach on both anatomy classification and image retrieval tasks using the Chest ImaGenome Dataset. Our proposed strategy yields an improvement over state-of-the-art pretraining and co-training strategies, from 92.24 to 94.12 (2.03%) classification accuracy in anatomies. We qualitatively evaluate the image retrieval performance demonstrating generalizability across multiple anatomies with different morphology.