# GMISeg: General Medical Image Segmentation without Re-Training

arXiv: https://arxiv.org/abs/2311.12539 · HTML: https://mlsanity.com/paper/2311.12539/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 13 of 13)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation](https://mlsanity.com/paper/1804.02967/) (2018)
- [Recurrent Mask Refinement for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2108.00622/) (2021)
- [Neuralizer: General Neuroimage Analysis without Re-Training](https://mlsanity.com/paper/2305.02644/) (2023)
- [Cheap Lunch for Medical Image Segmentation by Fine-tuning SAM on Few Exemplars](https://mlsanity.com/paper/2308.14133/) (2023)
- [Generalized Few-Shot Semantic Segmentation: All You Need is Fine-Tuning](https://mlsanity.com/paper/2112.10982/) (2021)
- [Task-Adaptive Feature Transformer with Semantic Enrichment for Few-Shot Segmentation](https://mlsanity.com/paper/2202.06498/) (2022)

## Abstract (arXiv)

Deep learning models have become the dominant method for medical image segmentation. However, they often struggle to be generalisable to unknown tasks involving new anatomical structures, labels, or shapes. In these cases, the model needs to be re-trained for the new tasks, posing a significant challenge for non-machine learning experts and requiring a considerable time investment. Here I developed a general model that can solve unknown medical image segmentation tasks without requiring additional training. Given an example set of images and visual prompts for defining new segmentation tasks, GMISeg (General Medical Image Segmentation) leverages a pre-trained image encoder based on ViT and applies a low-rank fine-tuning strategy to the prompt encoder and mask decoder to fine-tune the model without in an efficient manner. I evaluated the performance of the proposed method on medical image datasets with different imaging modalities and anatomical structures. The proposed method facilitated the deployment of pre-trained AI models to new segmentation works in a user-friendly way.