# PromptForSegCXR: Prompt-Driven Multi-Organ and Multi-Disease Segmentation in Chest X-rays using a Multi-stage Fusion Mechanism

arXiv: https://arxiv.org/abs/2507.00673 · HTML: https://mlsanity.com/paper/2507.00673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2211.14813/) (2022)
- [Brain Tumor Segmentation with Deep Neural Networks](https://mlsanity.com/paper/1505.03540/) (2015)
- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [CycleMix: A Holistic Strategy for Medical Image Segmentation from Scribble Supervision](https://mlsanity.com/paper/2203.01475/) (2022)
- [Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/1803.02315/) (2018)
- [VinDr-RibCXR: A Benchmark Dataset for Automatic Segmentation and Labeling of Individual Ribs on Chest X-rays](https://mlsanity.com/paper/2107.01327/) (2021)
- [Machine Learning Techniques for Biomedical Image Segmentation: An Overview of Technical Aspects and Introduction to State-of-Art Applications](https://mlsanity.com/paper/1911.02521/) (2019)

## Abstract (arXiv)

Image segmentation is central to automated medical image analysis, enabling precise identification of anatomical structures and pathological regions. Conventional segmentation models typically target a single organ or disease, limiting their adaptability across clinical scenarios. While multi-organ and multi-disease segmentation has been explored, building such datasets requires extensive manual annotation by medical experts. Prompt-driven segmentation offers a flexible, user-guided alternative that speeds up annotation, yet no prior work has addressed prompt-based interactive segmentation across multiple organs and diseases in chest X-rays. This study makes two main contributions. First, we introduce a novel dataset of expert-designed doodle prompts spanning 23 classes (six organs and seventeen diseases), curated from multiple public chest X-ray datasets for prompt-driven segmentation. Second, we propose PromptForSegCXR, a lightweight dual-input segmentation framework that combines the chest X-ray with user-provided doodle prompts to accurately segment diverse anatomical and pathological regions. The model uses a multi-stage feature fusion strategy to integrate spatial and semantic representations, along with a depthwise-pointwise-residual convolution block with squeeze-and-excitation attention for efficient hierarchical feature extraction and adaptive recalibration. Experimental results show the model achieves a Dice score of 81.62 percent on the full dataset, outperforming SAM-based prompt segmentation models by up to 10 percent and conventional segmentation architectures by up to 23 percent, while remaining lightweight. These results demonstrate the effectiveness of the proposed approach for accurate, flexible, prompt-driven chest X-ray segmentation.