# Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/1911.10334 · HTML: https://mlsanity.com/paper/1911.10334/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2019.

## Built on by (most cited first, 30 of 41)

- [Neural Volumetric Object Selection](https://mlsanity.com/paper/2205.14929/) (2022)
- [MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning](https://mlsanity.com/paper/2104.12166/) (2021)
- [Deep reinforcement learning in medical imaging: A literature review](https://mlsanity.com/paper/2103.05115/) (2021)
- [Artificial General Intelligence for Medical Imaging Analysis](https://mlsanity.com/paper/2306.05480/) (2023)
- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models](https://mlsanity.com/paper/2103.00429/) (2021)
- [Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.04148/) (2023)
- [Interactive 3D Medical Image Segmentation with SAM 2](https://mlsanity.com/paper/2408.02635/) (2024)
- [Reinforcement Learning in Medical Image Analysis: Concepts, Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2206.14302/) (2022)
- [Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directions](https://mlsanity.com/paper/2303.12484/) (2023)
- [Boundary-aware Supervoxel-level Iteratively Refined Interactive 3D Image Segmentation with Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2303.10692/) (2023)
- [Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2403.07939/) (2024)
- [Clustering Propagation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.16646/) (2024)
- [Temporally-Extended Prompts Optimization for SAM in Interactive Medical Image Segmentation](https://mlsanity.com/paper/2306.08958/) (2023)
- [Local Patch AutoAugment with Multi-Agent Collaboration](https://mlsanity.com/paper/2103.11099/) (2021)
- [Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization](https://mlsanity.com/paper/2311.02719/) (2023)
- [Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions](https://mlsanity.com/paper/2209.10307/) (2022)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Lane2Seq: Towards Unified Lane Detection via Sequence Generation](https://mlsanity.com/paper/2402.17172/) (2024)
- [Quality-Aware Memory Network for Interactive Volumetric Image Segmentation](https://mlsanity.com/paper/2106.10686/) (2021)
- [Interactive Medical Image Segmentation with Self-Adaptive Confidence Calibration](https://mlsanity.com/paper/2111.07716/) (2021)
- [Exploring Cycle Consistency Learning in Interactive Volume Segmentation](https://mlsanity.com/paper/2303.06493/) (2023)
- [Deep Reinforcement Learning Framework for Thoracic Diseases Classification via Prior Knowledge Guidance](https://mlsanity.com/paper/2306.01232/) (2023)
- [Soft-CP: A Credible and Effective Data Augmentation for Semantic Segmentation of Medical Lesions](https://mlsanity.com/paper/2203.10507/) (2022)
- [Rapid model transfer for medical image segmentation via iterative human-in-the-loop update: from labelled public to unlabelled clinical datasets for multi-organ segmentation in CT](https://mlsanity.com/paper/2204.06243/) (2022)
- [Learning Roles with Emergent Social Value Orientations](https://mlsanity.com/paper/2301.13812/) (2023)
- [Sequential Attention-based Sampling for Histopathological Analysis](https://mlsanity.com/paper/2507.05077/) (2025)
- [But that's not why: Inference adjustment by interactive prototype revision](https://mlsanity.com/paper/2203.10087/) (2022)
- [A dynamic interactive learning framework for automated 3D medical image segmentation](https://mlsanity.com/paper/2312.06072/) (2023)
- [SMAUG: A Sliding Multidimensional Task Window-Based MARL Framework for Adaptive Real-Time Subtask Recognition](https://mlsanity.com/paper/2403.01816/) (2024)

## Abstract (arXiv)

Existing automatic 3D image segmentation methods usually fail to meet the clinic use. Many studies have explored an interactive strategy to improve the image segmentation performance by iteratively incorporating user hints. However, the dynamic process for successive interactions is largely ignored. We here propose to model the dynamic process of iterative interactive image segmentation as a Markov decision process (MDP) and solve it with reinforcement learning (RL). Unfortunately, it is intractable to use single-agent RL for voxel-wise prediction due to the large exploration space. To reduce the exploration space to a tractable size, we treat each voxel as an agent with a shared voxel-level behavior strategy so that it can be solved with multi-agent reinforcement learning. An additional advantage of this multi-agent model is to capture the dependency among voxels for segmentation task. Meanwhile, to enrich the information of previous segmentations, we reserve the prediction uncertainty in the state space of MDP and derive an adjustment action space leading to a more precise and finer segmentation. In addition, to improve the efficiency of exploration, we design a relative cross-entropy gain-based reward to update the policy in a constrained direction. Experimental results on various medical datasets have shown that our method significantly outperforms existing state-of-the-art methods, with the advantage of fewer interactions and a faster convergence.