mlsanityData to 7 Oct 2026

Logit-Level Uncertainty Quantification in Vision-Language Models for Histopathology Image Analysis

arXiv 2603.03527 ↗First posted 3 Mar 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) with their multimodal capabilities have demonstrated remarkable success in almost all domains, including education, transportation, healthcare, energy, finance, law, and retail. Nevertheless, the utilization of VLMs in healthcare applications raises crucial concerns due to the sensitivity of large-scale medical data and the trustworthiness of these models (reliability, transparency, and security). This study proposes a logit-level uncertainty quantification (UQ) framework for histopathology image analysis using VLMs to deal with these concerns. UQ is evaluated for three VLMs using metrics derived from temperature-controlled output logits. The proposed framework demonstrates a critical separation in uncertainty behavior. While VLMs show high stochastic sensitivity (cosine similarity (CS) <0.71 and <0.84, Jensen-Shannon divergence (JS) <0.57 and <0.38, and Kullback-Leibler divergence (KL) <0.55 and <0.35, respectively for mean values of VILA-M3-8B and LLaVA-Med v1.5), near-maximal temperature impacts (DeltaT ≈ 1.00), and displaying abrupt uncertainty transitions, particularly for complex diagnostic prompts. In contrast, the pathology-specific PRISM model maintains near-deterministic behavior (mean CS >0.90, JS <0.10, KL <0.09) and significantly minimal temperature effects across all prompt complexities. These findings emphasize the importance of logit-level uncertainty quantification to evaluate trustworthiness in histopathology applications utilizing VLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 20 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

9 of 20 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis31 Oct
  2. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge19 Nov
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 6

  1. 2024An Introduction to Vision-Language Modelingcited by 175
  2. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
  3. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 28
  4. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Modelscited by 9
  5. 2025Lifelong Whole Slide Image Analysis: Online Vision-Language Adaptation and Past-to-Present Gradient Distillationcited by 2
  6. 2025Value-Guided Iterative Refinement and the DIQ-H Benchmark for Evaluating VLM Robustnesscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark
  2. 2026Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology
  3. 2025Effortless Vision-Language Model Specialization in Histopathology without Annotation
  4. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
  5. 2025MedVision: Benchmarking Quantitative Medical Image Analysis
  6. 2024Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysis
  7. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  8. 2025Distribution-Based Masked Medical Vision-Language Model Using Structured Reports