mlsanityData to 14 Oct 2026

Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context

arXiv 2506.12683 ↗First posted 15 Jun 2025cs.CV q-bio.QMcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) have rapidly advanced alongside Large Language Models (LLMs). This study evaluates the capabilities of prominent generative VLMs, such as GPT-4.1 and Gemini 2.5 Pro, accessed via APIs, for histopathology image classification tasks, including cell typing. Using diverse datasets from public and private sources, we apply zero-shot and one-shot prompting methods to assess VLM performance, comparing them against custom-trained Convolutional Neural Networks (CNNs). Our findings demonstrate that while one-shot prompting significantly improves VLM performance over zero-shot (p ≈ 1.005 × 10-5 based on Kappa scores), these general-purpose VLMs currently underperform supervised CNNs on most tasks. This work underscores both the promise and limitations of applying current VLMs to specialized domains like pathology via in-context learning. All code and instructions for reproducing the study can be accessed from the repository https://www.github.com/a12dongithub/VLMCCE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 21 references that are in the map. First posted on arXiv 15 Jun 2025.

Builds on

9 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024In-context learning enables multimodal large language models to classify cancer pathology images12 Mar
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 7

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2015An Introduction to Convolutional Neural Networkscited by 514
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2019Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Imagescited by 39

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Effortless Vision-Language Model Specialization in Histopathology without Annotation
  2. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  3. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
  4. 2024MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological images
  5. 2025Democratising Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modelling
  6. 2025Slide-Level Prompt Learning with Vision Language Models for Few-Shot Multiple Instance Learning in Histopathology
  7. 2025In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy
  8. 2025VLM-based Prompts as the Optimal Assistant for Unpaired Histopathology Virtual Staining