# Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline

arXiv: https://arxiv.org/abs/2411.12814 · HTML: https://mlsanity.com/paper/2411.12814/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 136 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 30 of 71)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [A large annotated medical image dataset for the development and evaluation of segmentation algorithms](https://mlsanity.com/paper/1902.09063/) (2019)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation](https://mlsanity.com/paper/2001.06535/) (2020)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge](https://mlsanity.com/paper/1612.08012/) (2016)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)

## Built on by (most cited first, 30 of 30)

- [BRISC: Annotated Dataset for Brain Tumor Segmentation and Classification](https://mlsanity.com/paper/2506.14318/) (2025)
- [MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance](https://mlsanity.com/paper/2412.15058/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generation](https://mlsanity.com/paper/2412.01762/) (2024)
- [Visual Instruction Pretraining for Domain-Specific Foundation Models](https://mlsanity.com/paper/2509.17562/) (2025)
- [MediSee: Reasoning-based Pixel-level Perception in Medical Images](https://mlsanity.com/paper/2504.11008/) (2025)
- [MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.03320/) (2026)
- [Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset](https://mlsanity.com/paper/2508.10528/) (2025)
- [Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2](https://mlsanity.com/paper/2505.01854/) (2025)
- [Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset](https://mlsanity.com/paper/2511.15186/) (2025)
- [UKBOB: One Billion MRI Labeled Masks for Generalizable 3D Medical Image Segmentation](https://mlsanity.com/paper/2504.06908/) (2025)
- [SAM-aware Test-time Adaptation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2506.05221/) (2025)
- [MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis](https://mlsanity.com/paper/2602.22955/) (2026)
- [MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives](https://mlsanity.com/paper/2501.04184/) (2025)
- [Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks](https://mlsanity.com/paper/2511.06665/) (2025)
- [Adaptive Frequency Domain Alignment Network for Medical image segmentation](https://mlsanity.com/paper/2512.16393/) (2025)
- [SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation](https://mlsanity.com/paper/2602.19213/) (2026)
- [MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images](https://mlsanity.com/paper/2511.12110/) (2025)
- [Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning](https://mlsanity.com/paper/2511.17201/) (2025)
- [Domain-Specialized Interactive Segmentation Framework for Meningioma Radiotherapy Planning](https://mlsanity.com/paper/2510.00416/) (2025)
- [Vision-Language Controlled Deep Unfolding for Joint Medical Image Restoration and Segmentation](https://mlsanity.com/paper/2601.23103/) (2026)
- [MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Training](https://mlsanity.com/paper/2604.10242/) (2026)
- [MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration](https://mlsanity.com/paper/2604.11197/) (2026)
- [MorphoFlow: Sparse-Supervised Generative Shape Modeling with Adaptive Latent Relevance](https://mlsanity.com/paper/2604.11636/) (2026)
- [Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty](https://mlsanity.com/paper/2606.28520/) (2026)
- [UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation](https://mlsanity.com/paper/2607.12896/) (2026)
- [ReportMedSAM: Guiding Segmentation Through Radiology Reports](https://mlsanity.com/paper/2607.14116/) (2026)
- [Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models](https://mlsanity.com/paper/2608.00976/) (2026)
- [EliSeg: Verified Target Construction for Report-Grounded Abnormality Segmentation](https://mlsanity.com/paper/2608.07299/) (2026)
- [XFlow: A Workflow Model for Instruction-Guided Lesion Segmentation in Chest X-rays](https://mlsanity.com/paper/2609.34513/) (2026)

## Abstract (arXiv)

Interactive Medical Image Segmentation (IMIS) has long been constrained by the limited availability of large-scale, diverse, and densely annotated datasets, which hinders model generalization and consistent evaluation across different models. In this paper, we introduce the IMed-361M benchmark dataset, a significant advancement in general IMIS research. First, we collect and standardize over 6.4 million medical images and their corresponding ground truth masks from multiple data sources. Then, leveraging the strong object recognition capabilities of a vision foundational model, we automatically generated dense interactive masks for each image and ensured their quality through rigorous quality control and granularity management. Unlike previous datasets, which are limited by specific modalities or sparse annotations, IMed-361M spans 14 modalities and 204 segmentation targets, totaling 361 million masks-an average of 56 masks per image. Finally, we developed an IMIS baseline network on this dataset that supports high-quality mask generation through interactive inputs, including clicks, bounding boxes, text prompts, and their combinations. We evaluate its performance on medical image segmentation tasks from multiple perspectives, demonstrating superior accuracy and scalability compared to existing interactive segmentation models. To facilitate research on foundational models in medical computer vision, we release the IMed-361M and model at https://github.com/uni-medical/IMIS-Bench.