# Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting

arXiv: https://arxiv.org/abs/2603.21287 · HTML: https://mlsanity.com/paper/2603.21287/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 61 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation](https://mlsanity.com/paper/2001.06535/) (2020)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [Self-Supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotation](https://mlsanity.com/paper/2007.09886/) (2020)
- [Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Counting](https://mlsanity.com/paper/2203.08354/) (2022)
- ['Squeeze & Excite' Guided Few-Shot Segmentation of Volumetric Images](https://mlsanity.com/paper/1902.01314/) (2019)
- [Recurrent Mask Refinement for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2108.00622/) (2021)
- [Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding](https://mlsanity.com/paper/2012.05440/) (2020)
- [Anomaly Detection-Inspired Few-Shot Medical Image Segmentation Through Self-Supervision With Supervoxels](https://mlsanity.com/paper/2203.02048/) (2022)
- [Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2308.07624/) (2023)
- [One-Shot Medical Landmark Detection](https://mlsanity.com/paper/2103.04527/) (2021)
- [Few-Shot Medical Image Segmentation via a Region-enhanced Prototypical Transformer](https://mlsanity.com/paper/2309.04825/) (2023)
- [Few-shot Keypoint Detection with Uncertainty Learning for Unseen Species](https://mlsanity.com/paper/2112.06183/) (2021)
- [Q-Net: Query-Informed Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2208.11451/) (2022)
- [Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation](https://mlsanity.com/paper/2307.03407/) (2023)
- [Few Shot Medical Image Segmentation with Cross Attention Transformer](https://mlsanity.com/paper/2303.13867/) (2023)
- [Self-Sampling Meta SAM: Enhancing Few-shot Medical Image Segmentation with Meta-Learning](https://mlsanity.com/paper/2308.16466/) (2023)
- [Which images to label for few-shot medical landmark detection?](https://mlsanity.com/paper/2112.04386/) (2021)
- [FAMNet: Frequency-aware Matching Network for Cross-domain Few-shot Medical Image Segmentation](https://mlsanity.com/paper/2412.09319/) (2024)
- [Prototype Correlation Matching and Class-Relation Reasoning for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2406.05054/) (2024)

## Built on by (most cited first, 1 of 1)

- [Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2607.27856/) (2026)

## Abstract (arXiv)

Conventional few-shot medical image segmentation (FSMIS) approaches face performance bottlenecks that hinder broader clinical applicability. Although the Segment Anything Model (SAM) exhibits strong category-agnostic segmentation capabilities, its direct application to medical images often leads to over-segmentation due to ambiguous anatomical boundaries. In this paper, we reformulate SAM-based FSMIS as a prompt localization task and propose FoB (Focus on Background), a background-centric prompt generator that provides accurate background prompts to constrain SAM's over-segmentation. Specifically, FoB bridges the gap between segmentation and prompt localization by category-agnostic generation of support background prompts and localizing them directly in the query image. To address the challenge of prompt localization for novel categories, FoB models rich contextual information to capture foreground-background spatial dependencies. Moreover, inspired by the inherent structural patterns of background prompts in medical images, FoB models this structure as a constraint to progressively refine background prompt predictions. Experiments on three diverse medical image datasets demonstrate that FoB outperforms other baselines by large margins, achieving state-of-the-art performance on FSMIS, and exhibiting strong cross-domain generalization. Our code is available at https://github.com/primebo1/FoB_SAM.