# 3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology

arXiv: https://arxiv.org/abs/2311.13752 · HTML: https://mlsanity.com/paper/2311.13752/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 23 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [3D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2209.15076/) (2022)
- [A Decade Survey of Content Based Image Retrieval using Deep Learning](https://mlsanity.com/paper/2012.00641/) (2020)
- [Medical Image Retrieval using Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.08472/) (2017)
- [A Survey on Biomedical Image Captioning](https://mlsanity.com/paper/1905.13302/) (2019)
- [Medical Image Captioning via Generative Pretrained Transformers](https://mlsanity.com/paper/2209.13983/) (2022)
- [Scaling Up 3D Kernels with Bayesian Frequency Re-parameterization for Medical Image Segmentation](https://mlsanity.com/paper/2303.05785/) (2023)
- [Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query](https://mlsanity.com/paper/2305.05598/) (2023)

## Built on by (most cited first, 12 of 12)

- [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://mlsanity.com/paper/2410.06542/) (2024)
- [BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrieval](https://mlsanity.com/paper/2403.15992/) (2024)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Leveraging Foundation Models for Content-Based Image Retrieval in Radiology](https://mlsanity.com/paper/2403.06567/) (2024)
- [M3Retrieve: Benchmarking Multimodal Retrieval for Medicine](https://mlsanity.com/paper/2510.06888/) (2025)
- [Content-Based Image Retrieval for Multi-Class Volumetric Radiology Images: A Benchmark Study](https://mlsanity.com/paper/2405.09334/) (2024)
- [RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining](https://mlsanity.com/paper/2503.04653/) (2025)
- [M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision](https://mlsanity.com/paper/2509.01360/) (2025)
- [RadiomicsRetrieval: A Customizable Framework for Medical Image Retrieval Using Radiomics Features](https://mlsanity.com/paper/2507.08546/) (2025)
- [Content-based 3D Image Retrieval and a ColBERT-inspired Re-ranking for Tumor Flagging and Staging](https://mlsanity.com/paper/2507.17412/) (2025)
- [WristMIR: Coarse-to-Fine Region-Aware Retrieval of Pediatric Wrist Radiographs with Radiology Report-Driven Learning](https://mlsanity.com/paper/2602.07872/) (2026)

## Abstract (arXiv)

The increasing use of medical imaging in healthcare settings presents a significant challenge due to the increasing workload for radiologists, yet it also offers opportunity for enhancing healthcare outcomes if effectively leveraged. 3D image retrieval holds potential to reduce radiologist workloads by enabling clinicians to efficiently search through diagnostically similar or otherwise relevant cases, resulting in faster and more precise diagnoses. However, the field of 3D medical image retrieval is still emerging, lacking established evaluation benchmarks, comprehensive datasets, and thorough studies. This paper attempts to bridge this gap by introducing a novel benchmark for 3D Medical Image Retrieval (3D-MIR) that encompasses four different anatomies imaged with computed tomography. Using this benchmark, we explore a diverse set of search strategies that use aggregated 2D slices, 3D volumes, and multi-modal embeddings from popular multi-modal foundation models as queries. Quantitative and qualitative assessments of each approach are provided alongside an in-depth discussion that offers insight for future research. To promote the advancement of this field, our benchmark, dataset, and code are made publicly available.