# MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning

arXiv: https://arxiv.org/abs/2104.12166 · HTML: https://mlsanity.com/paper/2104.12166/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 25 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?](https://mlsanity.com/paper/1706.00712/) (2017)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [Bottom-up Object Detection by Grouping Extreme and Center Points](https://mlsanity.com/paper/1901.08043/) (2019)
- [Deep Extreme Cut: From Extreme Points to Object Segmentation](https://mlsanity.com/paper/1711.09081/) (2017)
- [Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++](https://mlsanity.com/paper/1803.09693/) (2018)
- [Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning](https://mlsanity.com/paper/1710.04043/) (2017)
- [DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation](https://mlsanity.com/paper/1707.00652/) (2017)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [Annotating Object Instances with a Polygon-RNN](https://mlsanity.com/paper/1704.05548/) (2017)
- [Extreme clicking for efficient object annotation](https://mlsanity.com/paper/1708.02750/) (2017)
- [DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks](https://mlsanity.com/paper/1605.07866/) (2016)
- [Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.10334/) (2019)
- [Going to Extremes: Weakly Supervised Medical Image Segmentation](https://mlsanity.com/paper/2009.11988/) (2020)
- [Deep GrabCut for Object Selection](https://mlsanity.com/paper/1707.00243/) (2017)
- [Interactive segmentation of medical images through fully convolutional neural networks](https://mlsanity.com/paper/1903.08205/) (2019)
- [Guide Me: Interacting with Deep Networks](https://mlsanity.com/paper/1803.11544/) (2018)
- [Uncertainty-Guided Efficient Interactive Refinement of Fetal Brain Segmentation from Stacks of MRI Slices](https://mlsanity.com/paper/2007.00833/) (2020)
- [A Fully Convolutional Two-Stream Fusion Network for Interactive Image Segmentation](https://mlsanity.com/paper/1807.02480/) (2018)
- [Weakly supervised segmentation from extreme points](https://mlsanity.com/paper/1910.01236/) (2019)
- [User-Guided Domain Adaptation for Rapid Annotation from User Interactions: A Study on Pathological Liver Segmentation](https://mlsanity.com/paper/2009.02455/) (2020)

## Built on by (most cited first, 30 of 45)

- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](https://mlsanity.com/paper/2310.15161/) (2023)
- [Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer](https://mlsanity.com/paper/2112.04894/) (2021)
- [3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation](https://mlsanity.com/paper/2306.13465/) (2023)
- [Scribble-Supervised Medical Image Segmentation via Dual-Branch Network and Dynamically Mixed Pseudo Labels Supervision](https://mlsanity.com/paper/2203.02106/) (2022)
- [ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image](https://mlsanity.com/paper/2312.07381/) (2023)
- [MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images](https://mlsanity.com/paper/2203.12362/) (2022)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [SegRap2023: A Benchmark of Organs-at-Risk and Gross Tumor Volume Segmentation for Radiotherapy Planning of Nasopharyngeal Carcinoma](https://mlsanity.com/paper/2312.09576/) (2023)
- [iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images](https://mlsanity.com/paper/2112.11325/) (2021)
- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [SATS: Self-Attention Transfer for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.07667/) (2022)
- [PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation](https://mlsanity.com/paper/2208.05669/) (2022)
- [Challenges for machine learning in clinical translation of big data imaging studies](https://mlsanity.com/paper/2107.05630/) (2021)
- [PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts](https://mlsanity.com/paper/2404.15028/) (2024)
- [mathrmSAMMed: A medical image annotation framework based on large vision model](https://mlsanity.com/paper/2307.05617/) (2023)
- [MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance](https://mlsanity.com/paper/2412.15058/) (2024)
- [Clustering Propagation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.16646/) (2024)
- [ECONet: Efficient Convolutional Online Likelihood Network for Scribble-based Interactive Segmentation](https://mlsanity.com/paper/2201.04584/) (2022)
- [Transforming the Interactive Segmentation for Medical Imaging](https://mlsanity.com/paper/2208.09592/) (2022)
- [Beyond Full Labels: Energy-Double-Guided Single-Point Prompt for Infrared Small Target Label Generation](https://mlsanity.com/paper/2408.08191/) (2024)
- [Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images based on Physiological and Anatomical Cues](https://mlsanity.com/paper/2301.09914/) (2023)
- [VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis](https://mlsanity.com/paper/2410.08397/) (2024)
- [Guiding the Guidance: A Comparative Analysis of User Guidance Signals for Interactive Segmentation of Volumetric Images](https://mlsanity.com/paper/2303.06942/) (2023)
- [MedUHIP: Towards Human-In-the-Loop Medical Segmentation](https://mlsanity.com/paper/2408.01620/) (2024)
- [Diversity-Promoting Ensemble for Medical Image Segmentation](https://mlsanity.com/paper/2210.12388/) (2022)
- [Adaptive Multi-scale Online Likelihood Network for AI-assisted Interactive Segmentation](https://mlsanity.com/paper/2303.13696/) (2023)
- [UM-CAM: Uncertainty-weighted Multi-resolution Class Activation Maps for Weakly-supervised Fetal Brain Segmentation](https://mlsanity.com/paper/2306.11490/) (2023)
- [Hybrid Dual Mean-Teacher Network With Double-Uncertainty Guidance for Semi-Supervised Segmentation of MRI Scans](https://mlsanity.com/paper/2303.05126/) (2023)
- [Rethinking Annotator Simulation: Realistic Evaluation of Whole-Body PET Lesion Interactive Segmentation Methods](https://mlsanity.com/paper/2404.01816/) (2024)

## Abstract (arXiv)

Segmentation of organs or lesions from medical images plays an essential role in many clinical applications such as diagnosis and treatment planning. Though Convolutional Neural Networks (CNN) have achieved the state-of-the-art performance for automatic segmentation, they are often limited by the lack of clinically acceptable accuracy and robustness in complex cases. Therefore, interactive segmentation is a practical alternative to these methods. However, traditional interactive segmentation methods require a large amount of user interactions, and recently proposed CNN-based interactive segmentation methods are limited by poor performance on previously unseen objects. To solve these problems, we propose a novel deep learning-based interactive segmentation method that not only has high efficiency due to only requiring clicks as user inputs but also generalizes well to a range of previously unseen objects. Specifically, we first encode user-provided interior margin points via our proposed exponentialized geodesic distance that enables a CNN to achieve a good initial segmentation result of both previously seen and unseen objects, then we use a novel information fusion method that combines the initial segmentation with only few additional user clicks to efficiently obtain a refined segmentation. We validated our proposed framework through extensive experiments on 2D and 3D medical image segmentation tasks with a wide range of previous unseen objects that were not present in the training set. Experimental results showed that our proposed framework 1) achieves accurate results with fewer user interactions and less time compared with state-of-the-art interactive frameworks and 2) generalizes well to previously unseen objects.