# SeqSAM: Autoregressive Multiple Hypothesis Prediction for Medical Image Segmentation using SAM

arXiv: https://arxiv.org/abs/2503.09797 · HTML: https://mlsanity.com/paper/2503.09797/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 13 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 11 of 11)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Ambiguous Medical Image Segmentation using Diffusion Models](https://mlsanity.com/paper/2304.04745/) (2023)
- [Deep Set Prediction Networks](https://mlsanity.com/paper/1906.06565/) (2019)
- [QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge](https://mlsanity.com/paper/2405.18435/) (2024)

## Built on by (most cited first, 1 of 1)

- [SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM](https://mlsanity.com/paper/2609.18037/) (2026)

## Abstract (arXiv)

Pre-trained segmentation models are a powerful and flexible tool for segmenting images. Recently, this trend has extended to medical imaging. Yet, often these methods only produce a single prediction for a given image, neglecting inherent uncertainty in medical images, due to unclear object boundaries and errors caused by the annotation tool. Multiple Choice Learning is a technique for generating multiple masks, through multiple learned prediction heads. However, this cannot readily be extended to producing more outputs than its initial pre-training hyperparameters, as the sparse, winner-takes-all loss function makes it easy for one prediction head to become overly dominant, thus not guaranteeing the clinical relevancy of each mask produced. We introduce SeqSAM, a sequential, RNN-inspired approach to generating multiple masks, which uses a bipartite matching loss for ensuring the clinical relevancy of each mask, and can produce an arbitrary number of masks. We show notable improvements in quality of each mask produced across two publicly available datasets. Our code is available at https://github.com/BenjaminTowle/SeqSAM.