# PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation

arXiv: https://arxiv.org/abs/2503.18227 · HTML: https://mlsanity.com/paper/2503.18227/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything](https://mlsanity.com/paper/2304.06022/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More](https://mlsanity.com/paper/2304.09148/) (2023)
- [How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images](https://mlsanity.com/paper/2306.13731/) (2023)
- [Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding](https://mlsanity.com/paper/2403.18271/) (2024)
- [RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features](https://mlsanity.com/paper/2104.08569/) (2021)
- [Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets](https://mlsanity.com/paper/2304.09324/) (2023)
- [TransDeepLab: Convolution-Free Transformer-based DeepLab v3+ for Medical Image Segmentation](https://mlsanity.com/paper/2208.00713/) (2022)
- [Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2303.16892/) (2023)
- [SkinSAM: Empowering Skin Cancer Segmentation with Segment Anything Model](https://mlsanity.com/paper/2304.13973/) (2023)
- [DAE-Former: Dual Attention-guided Efficient Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2212.13504/) (2022)
- [Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation](https://mlsanity.com/paper/2404.06362/) (2024)
- [UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images](https://mlsanity.com/paper/2402.16663/) (2024)
- [Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few Exemplars](https://mlsanity.com/paper/2308.14133/) (2023)
- [Self-Sampling Meta SAM: Enhancing Few-shot Medical Image Segmentation with Meta-Learning](https://mlsanity.com/paper/2308.16466/) (2023)
- [Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2403.07630/) (2024)
- [TP-DRSeg: Improving Diabetic Retinopathy Lesion Segmentation with Explicit Text-Prompts Assisted SAM](https://mlsanity.com/paper/2406.15764/) (2024)
- [Aligning Medical Images with General Knowledge from Large Language Models](https://mlsanity.com/paper/2409.00341/) (2024)
- [MSWAL: 3D Multi-class Segmentation of Whole Abdominal Lesions Dataset](https://mlsanity.com/paper/2503.13560/) (2025)
- [SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation](https://mlsanity.com/paper/2412.12660/) (2024)

## Built on by (most cited first, 2 of 2)

- [SAM-DCE: Addressing Token Uniformity and Semantic Over-Smoothing in Medical Segmentation](https://mlsanity.com/paper/2509.16886/) (2025)
- [SPG-CDENet: Spatial Prior-Guided Cross Dual Encoder Network for Multi-Organ Segmentation](https://mlsanity.com/paper/2510.26390/) (2025)

## Abstract (arXiv)

Segment Anything Model (SAM) demonstrates powerful zero-shot capabilities; however, its accuracy and robustness significantly decrease when applied to medical image segmentation. Existing methods address this issue through modality fusion, integrating textual and image information to provide more detailed priors. In this study, we argue that the granularity of text and the domain gap affect the accuracy of the priors. Furthermore, the discrepancy between high-level abstract semantics and pixel-level boundary details in images can introduce noise into the fusion process. To address this, we propose Prior-Guided SAM (PG-SAM), which employs a fine-grained modality prior aligner to leverage specialized medical knowledge for better modality alignment. The core of our method lies in efficiently addressing the domain gap with fine-grained text from a medical LLM. Meanwhile, it also enhances the priors' quality after modality alignment, ensuring more accurate segmentation. In addition, our decoder enhances the model's expressive capabilities through multi-level feature fusion and iterative mask optimizer operations, supporting unprompted learning. We also propose a unified pipeline that effectively supplies high-quality semantic information to SAM. Extensive experiments on the Synapse dataset demonstrate that the proposed PG-SAM achieves state-of-the-art performance. Our code is released at https://github.com/logan-0623/PG-SAM.