mlsanityData to 14 Oct 2026

Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design

arXiv 2505.00134 ↗First posted 30 Apr 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have gained significant attention in computational pathology due to their multimodal learning capabilities that enhance big-data analytics of giga-pixel whole slide image (WSI). However, their sensitivity to large-scale clinical data, task formulations, and prompt design remains an open question, particularly in terms of diagnostic accuracy. In this paper, we present a systematic investigation and analysis of three state of the art VLMs for histopathology, namely Quilt-Net, Quilt-LLAVA, and CONCH, on an in-house digestive pathology dataset comprising 3,507 WSIs, each in giga-pixel form, across distinct tissue types. Through a structured ablative study on cancer invasiveness and dysplasia status, we develop a comprehensive prompt engineering framework that systematically varies domain specificity, anatomical precision, instructional framing, and output constraints. Our findings demonstrate that prompt engineering significantly impacts model performance, with the CONCH model achieving the highest accuracy when provided with precise anatomical references. Additionally, we identify the critical importance of anatomical context in histopathological image analysis, as performance consistently degraded when reducing anatomical precision. We also show that model complexity alone does not guarantee superior performance, as effective domain alignment and domain-specific training are critical. These results establish foundational guidelines for prompt engineering in computational pathology and highlight the potential of VLMs to enhance diagnostic accuracy when properly instructed with domain-appropriate prompts.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 29 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

21 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models23 Aug
  2. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding29 May
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  10. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  11. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134
  12. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 90
6 more
  1. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  2. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  3. 2023Computational Pathology: A Survey Review and The Way Forwardcited by 30
  4. 2024Task Supportive and Personalized Human-Large Language Model Interaction: A User Studycited by 17
  5. 2024IPO: Interpretable Prompt Optimization for Vision-Language Modelscited by 14
  6. 2023Towards Robust Prompts on Vision-Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Zero-shot segmentation of skin tumors in whole-slide images with vision-language foundation modelscited by 0
  2. 2026Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generationcited by 0
  3. 2026Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathologycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Effortless Vision-Language Model Specialization in Histopathology without Annotation
  2. 2025Slide-Level Prompt Learning with Vision Language Models for Few-Shot Multiple Instance Learning in Histopathology
  3. 2025Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
  4. 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
  5. 2025Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images
  6. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  7. 2025Democratising Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modelling
  8. 2024PathAlign: A vision-language model for whole slide images in histopathology