# ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax arXiv: https://arxiv.org/abs/2303.01615 · HTML: https://mlsanity.com/paper/2303.01615/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 20 of its 36 references that are in the map. - First posted on arXiv 2 Mar 2023. ## Placed in - Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/) ## Builds on (most cited first, 20 of 20) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014) - [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022) - [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020) - [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018) - [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017) - [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021) - [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022) - [Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports](https://mlsanity.com/paper/2110.03094/) (2021) - [Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation](https://mlsanity.com/paper/2009.13815/) (2020) - [Word2Pix: Word to Pixel Cross Attention Transformer in Visual Grounding](https://mlsanity.com/paper/2108.00205/) (2021) - [Content-Based Medical Image Retrieval with Opponent Class Adaptive Margin Loss](https://mlsanity.com/paper/2211.15371/) (2022) ## Built on by (most cited first, 10 of 10) - [LLM-driven Multimodal Target Volume Contouring in Radiation Oncology](https://mlsanity.com/paper/2311.01908/) (2023) - [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025) - [STPNet: Scale-aware Text Prompt Network for Medical Image Segmentation](https://mlsanity.com/paper/2504.01561/) (2025) - [Mixture of Multicenter Experts in Multimodal AI for Debiased Radiotherapy Target Delineation](https://mlsanity.com/paper/2410.00046/) (2024) - [BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation](https://mlsanity.com/paper/2503.23534/) (2025) - [Evidential learning driven Breast Tumor Segmentation with Stage-divided Vision-Language Interaction](https://mlsanity.com/paper/2603.11206/) (2026) - [Deep Learning for Longitudinal Gross Tumor Volume Segmentation in MRI-Guided Adaptive Radiotherapy for Head and Neck Cancer](https://mlsanity.com/paper/2412.00663/) (2024) - [Ontology-based knowledge representation for bone disease diagnosis: a foundation for safe and sustainable medical artificial intelligence systems](https://mlsanity.com/paper/2506.04756/) (2025) - [Knowledge-Guided Brain Tumor Segmentation via Synchronized Visual-Semantic-Topological Prior Fusion](https://mlsanity.com/paper/2507.08574/) (2025) - [MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation](https://mlsanity.com/paper/2608.13690/) (2026) ## Abstract (arXiv) Radiology narrative reports often describe characteristics of a patient's disease, including its location, size, and shape. Motivated by the recent success of multimodal learning, we hypothesized that this descriptive text could guide medical image analysis algorithms. We proposed a novel vision-language model, ConTEXTual Net, for the task of pneumothorax segmentation on chest radiographs. ConTEXTual Net utilizes language features extracted from corresponding free-form radiology reports using a pre-trained language model. Cross-attention modules are designed to combine the intermediate output of each vision encoder layer and the text embeddings generated by the language model. ConTEXTual Net was trained on the CANDID-PTX dataset consisting of 3,196 positive cases of pneumothorax with segmentation annotations from 6 different physicians as well as clinical radiology reports. Using cross-validation, ConTEXTual Net achieved a Dice score of 0.716±0.016, which was similar to the degree of inter-reader variability (0.712±0.044) computed on a subset of the data. It outperformed both vision-only models (ResNet50 U-Net: 0.677±0.015 and GLoRIA: 0.686±0.014) and a competing vision-language model (LAVT: 0.706±0.009). Ablation studies confirmed that it was the text information that led to the performance gains. Additionally, we show that certain augmentation methods degraded ConTEXTual Net's segmentation performance by breaking the image-text concordance. We also evaluated the effects of using different language models and activation functions in the cross-attention module, highlighting the efficacy of our chosen architectural design.