# SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidance

arXiv: https://arxiv.org/abs/2409.04758 · HTML: https://mlsanity.com/paper/2409.04758/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 7 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [U-Net and its variants for medical image segmentation: theory and applications](https://mlsanity.com/paper/2011.01118/) (2020)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practice](https://mlsanity.com/paper/1911.01685/) (2019)
- [OSegNet: Operational Segmentation Network for COVID-19 Detection using Chest X-ray Images](https://mlsanity.com/paper/2202.10185/) (2022)
- [Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray images](https://mlsanity.com/paper/2307.03942/) (2023)

## Built on by (most cited first, 11 of 11)

- [Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments](https://mlsanity.com/paper/2412.13533/) (2024)
- [Alleviating Textual Reliance in Medical Language-guided Segmentation via Prototype-driven Semantic Approximation](https://mlsanity.com/paper/2507.11055/) (2025)
- [Frequency-domain Multi-modal Fusion for Language-guided Medical Image Segmentation](https://mlsanity.com/paper/2509.19719/) (2025)
- [A Text-Image Fusion Method with Data Augmentation Capabilities for Referring Medical Image Segmentation](https://mlsanity.com/paper/2510.12482/) (2025)
- [ART-ASyn: Anatomy-aware Realistic Texture-based Anomaly Synthesis Framework for Chest X-Rays](https://mlsanity.com/paper/2512.00310/) (2025)
- [Localization-Infused Vision-Language Semantic Fusion for Text-Guided Medical Image Segmentation](https://mlsanity.com/paper/2607.16327/) (2026)
- [Toward Next-generation Medical Vision Backbones: Modeling Finer-grained Long-range Visual Dependency](https://mlsanity.com/paper/2509.11328/) (2025)
- [Medical Referring Image Segmentation via Next-Token Mask Prediction](https://mlsanity.com/paper/2511.05044/) (2025)
- [Semi-MedRef: Semi-Supervised Medical Referring Image Segmentation with Cross-Modal Alignment](https://mlsanity.com/paper/2605.15720/) (2026)
- [Instance-Guided Report Anchoring for Text-Free 3D Abnormality Segmentation in Chest CT](https://mlsanity.com/paper/2609.00447/) (2026)
- [CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation](https://mlsanity.com/paper/2609.04781/) (2026)

## Abstract (arXiv)

Segmentation of infected areas in chest X-rays is pivotal for facilitating the accurate delineation of pulmonary structures and pathological anomalies. Recently, multi-modal language-guided image segmentation methods have emerged as a promising solution for chest X-rays where the clinical text reports, depicting the assessment of the images, are used as guidance. Nevertheless, existing language-guided methods require clinical reports alongside the images, and hence, they are not applicable for use in image segmentation in a decision support context, but rather limited to retrospective image analysis after clinical reporting has been completed. In this study, we propose a self-guided segmentation framework (SGSeg) that leverages language guidance for training (multi-modal) while enabling text-free inference (uni-modal), which is the first that enables text-free inference in language-guided segmentation. We exploit the critical location information of both pulmonary and pathological structures depicted in the text reports and introduce a novel localization-enhanced report generation (LERG) module to generate clinical reports for self-guidance. Our LERG integrates an object detector and a location-based attention aggregator, weakly-supervised by a location-aware pseudo-label extraction module. Extensive experiments on a well-benchmarked QaTa-COV19 dataset demonstrate that our SGSeg achieved superior performance than existing uni-modal segmentation methods and closely matched the state-of-the-art performance of multi-modal language-guided segmentation methods.