# SimTxtSeg: Weakly-Supervised Medical Image Segmentation with Simple Text Cues

arXiv: https://arxiv.org/abs/2406.19364 · HTML: https://mlsanity.com/paper/2406.19364/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm](https://mlsanity.com/paper/1906.03720/) (2019)
- [A multi-centre polyp detection and segmentation dataset for generalisability assessment](https://mlsanity.com/paper/2106.04463/) (2021)
- [SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million masks](https://mlsanity.com/paper/2311.11969/) (2023)
- [Going to Extremes: Weakly Supervised Medical Image Segmentation](https://mlsanity.com/paper/2009.11988/) (2020)
- [Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray images](https://mlsanity.com/paper/2307.03942/) (2023)
- [ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding](https://mlsanity.com/paper/2307.16226/) (2023)
- [WeakPolyp: You Only Look Bounding Box for Polyp Segmentation](https://mlsanity.com/paper/2307.10912/) (2023)
- [S2ME: Spatial-Spectral Mutual Teaching and Ensemble Learning for Scribble-supervised Polyp Segmentation](https://mlsanity.com/paper/2306.00451/) (2023)
- [BoxPolyp:Boost Generalized Polyp Segmentation Using Extra Coarse Bounding Box Annotations](https://mlsanity.com/paper/2212.03498/) (2022)
- [Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation](https://mlsanity.com/paper/2306.03878/) (2023)
- [Learning to Segment Medical Images from Few-Shot Sparse Labels](https://mlsanity.com/paper/2108.05476/) (2021)
- [Box-Adapt: Domain-Adaptive Medical Image Segmentation using Bounding BoxSupervision](https://mlsanity.com/paper/2108.08432/) (2021)
- [BoxShrink: From Bounding Boxes to Segmentation Masks](https://mlsanity.com/paper/2208.03142/) (2022)
- [Boosting Weakly-Supervised Image Segmentation via Representation, Transform, and Compensator](https://mlsanity.com/paper/2309.00871/) (2023)

## Built on by (most cited first, 6 of 6)

- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Text-guided multi-stage cross-perception network for medical image segmentation](https://mlsanity.com/paper/2506.07475/) (2025)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis](https://mlsanity.com/paper/2605.18130/) (2026)
- [Beyond Visual Cues: CoT-Enhanced Reasoning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2606.17958/) (2026)
- [Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation](https://mlsanity.com/paper/2607.17341/) (2026)

## Abstract (arXiv)

Weakly-supervised medical image segmentation is a challenging task that aims to reduce the annotation cost while keep the segmentation performance. In this paper, we present a novel framework, SimTxtSeg, that leverages simple text cues to generate high-quality pseudo-labels and study the cross-modal fusion in training segmentation models, simultaneously. Our contribution consists of two key components: an effective Textual-to-Visual Cue Converter that produces visual prompts from text prompts on medical images, and a text-guided segmentation model with Text-Vision Hybrid Attention that fuses text and image features. We evaluate our framework on two medical image segmentation tasks: colonic polyp segmentation and MRI brain tumor segmentation, and achieve consistent state-of-the-art performance. Source code is available at: https://github.com/xyx1024/SimTxtSeg.