mlsanityData to 7 Oct 2026

Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation

arXiv 2605.01266 ↗First posted 2 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Zero-shot vision-language models (VLMs) offer a promptable alternative to task-specific training for gross tumor volume (GTV) delineation in non-small-cell lung cancer (NSCLC), but the prompt dimensions that govern their spatial behavior remain poorly understood. We study this question by probing alignment directions in VoxTell on a held-out internal NSCLC tumor dataset through sub-prompt decomposition into diagnosis, demographic, staging, anatomical, generic, and irrelevant controls; attribute-wise perturbation robustness; specificity ladders; and cross-case prompt swaps, while benchmarking against fine-tuned and zero-shot baselines using the Dice Similarity Coefficient (DSC) with Wilcoxon signed-rank tests and Benjamini-Hochberg correction. Alignment analyses revealed that anatomical location is the dominant driver of VoxTell's spatial attention: 63.4 percent of location perturbations caused catastrophic drops, prompt specificity improved from generic to full descriptions except for diagnosis-only prompts, irrelevant prompts correctly yielded zero segmentation, and cross-case prompt swaps confirmed patient-specific conditioning (matched DSC 0.906 vs. mismatched 0.406). Histology and stage substitutions had minimal effect, indicating that the model prioritizes "where to look" over "what to look for." In this context, VoxTell, operating fully zero-shot, achieved a mean DSC of 0.613, statistically indistinguishable from nnUNet (0.690, adjusted p = 0.156) and Ahmed et al. (0.675, adjusted p = 0.679), while significantly outperforming all other zero-shot models. Together, these findings argue that segmentation VLMs should be evaluated not only by Dice, but also by the prompt dimensions to which they align.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 12 references that are in the map. First posted on arXiv 2 May 2026.

Builds on

6 of 12 references.

References · 6

  1. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 597
  2. 2024VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imagingcited by 62
  3. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  4. 2023LLM-driven Multimodal Target Volume Contouring in Radiation Oncologycited by 20
  5. 2024CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentationcited by 14
  6. 2025Can Foundation Models Really Segment Tumors? A Benchmarking Odyssey in Lung CT Imagingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
  2. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification
  3. 2026ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification
  4. 2025Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
  5. 2026CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling
  6. 2026Learning How Much, Not Just What: Cross-Patient Burden Order for CT Vision-Language Pretraining
  7. 2024Lung-CADex: Fully automatic Zero-Shot Detection and Classification of Lung Nodules in Thoracic CT Images
  8. 2025XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography