# Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines

arXiv: https://arxiv.org/abs/2305.00109 · HTML: https://mlsanity.com/paper/2305.00109/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders](https://mlsanity.com/paper/2212.07050/) (2022)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [Zero-Shot and Few-Shot Learning for Lung Cancer Multi-Label Classification using Vision Transformer](https://mlsanity.com/paper/2205.15290/) (2022)

## Built on by (most cited first, 15 of 15)

- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [Foundation Models for Biomedical Image Segmentation: A Survey](https://mlsanity.com/paper/2401.07654/) (2024)
- [TongueSAM: An Universal Tongue Segmentation Model Based on SAM with Zero-Shot](https://mlsanity.com/paper/2308.06444/) (2023)
- [Curriculum Prompting Foundation Models for Medical Image Segmentation](https://mlsanity.com/paper/2409.00695/) (2024)
- [Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaul](https://mlsanity.com/paper/2307.12674/) (2023)
- [Exploring SAM Ablations for Enhancing Medical Segmentation in Radiology and Pathology](https://mlsanity.com/paper/2310.00504/) (2023)
- [Auto-Generating Weak Labels for Real & Synthetic Data to Improve Label-Scarce Medical Image Segmentation](https://mlsanity.com/paper/2404.17033/) (2024)
- [Zero-shot capability of SAM-family models for bone segmentation in CT scans](https://mlsanity.com/paper/2411.08629/) (2024)
- [SimSAM: Zero-shot Medical Image Segmentation via Simulated Interaction](https://mlsanity.com/paper/2406.00663/) (2024)
- [Medical Image Segmentation with SAM-generated Annotations](https://mlsanity.com/paper/2409.20253/) (2024)
- [MedFocusCLIP : Improving few shot classification in medical datasets using pixel wise attention](https://mlsanity.com/paper/2501.03839/) (2025)
- [Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare](https://mlsanity.com/paper/2505.08818/) (2025)
- [Comparative Analysis of Image Enhancement Techniques for Brain Tumor Segmentation: Contrast, Histogram, and Hybrid Approaches](https://mlsanity.com/paper/2404.05341/) (2024)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation](https://mlsanity.com/paper/2603.10541/) (2026)

## Abstract (arXiv)

Segmentation in medical imaging is a critical component for the diagnosis, monitoring, and treatment of various diseases and medical conditions. Presently, the medical segmentation landscape is dominated by numerous specialized deep learning models, each fine-tuned for specific segmentation tasks and image modalities. The recently-introduced Segment Anything Model (SAM) employs the ViT neural architecture and harnesses a massive training dataset to segment nearly any object; however, its suitability to the medical domain has not yet been investigated. In this study, we explore the zero-shot performance of SAM in medical imaging by implementing eight distinct prompt strategies across six datasets from four imaging modalities, including X-ray, ultrasound, dermatoscopy, and colonoscopy. Our findings reveal that SAM's zero-shot performance is not only comparable to, but in certain cases, surpasses the current state-of-the-art. Based on these results, we propose practical guidelines that require minimal interaction while consistently yielding robust outcomes across all assessed contexts. The source code, along with a demonstration of the recommended guidelines, can be accessed at https://github.com/Malta-Lab/SAM-zero-shot-in-Medical-Imaging.