# How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark

arXiv: https://arxiv.org/abs/2503.12990 · HTML: https://mlsanity.com/paper/2503.12990/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Gland Segmentation in Colon Histology Images: The GlaS Challenge Contest](https://mlsanity.com/paper/1603.00275/) (2016)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images](https://mlsanity.com/paper/2111.04740/) (2021)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [Lung and Colon Cancer Histopathological Image Dataset (LC25000)](https://mlsanity.com/paper/1912.12142/) (2019)
- [CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentation](https://mlsanity.com/paper/1908.10555/) (2019)
- [Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detection](https://mlsanity.com/paper/2105.10490/) (2021)
- [A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models](https://mlsanity.com/paper/2407.06508/) (2024)
- [A Petri Dish for Histopathology Image Analysis](https://mlsanity.com/paper/2101.12355/) (2021)
- [Benchmarking foundation models as feature extractors for weakly-supervised computational pathology](https://mlsanity.com/paper/2408.15823/) (2024)
- [Hard Sample Aware Noise Robust Learning for Histopathology Image Classification](https://mlsanity.com/paper/2112.03694/) (2021)
- [Multi-Layer Pseudo-Supervision for Histopathology Tissue Semantic Segmentation using Patch-level Classification Labels](https://mlsanity.com/paper/2110.08048/) (2021)
- [WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinoma](https://mlsanity.com/paper/2204.06455/) (2022)
- [Inference of captions from histopathological patches](https://mlsanity.com/paper/2202.03432/) (2022)
- [Impact of Stain Variation and Color Normalization for Prognostic Predictions in Pathology](https://mlsanity.com/paper/2409.08338/) (2024)
- [CRC-ICM: Colorectal Cancer Immune Cell Markers Pattern Dataset](https://mlsanity.com/paper/2308.10033/) (2023)

## Built on by (most cited first, 8 of 8)

- [PathBench: A comprehensive comparison benchmark for pathology foundation models towards precision oncology](https://mlsanity.com/paper/2505.20202/) (2025)
- [TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning](https://mlsanity.com/paper/2512.21331/) (2025)
- [Histopath-C: Towards Realistic Domain Shifts for Histopathology Vision-Language Adaptation](https://mlsanity.com/paper/2601.12493/) (2026)
- [Do Pathology Vision-Language Models Truly See Pathology?](https://mlsanity.com/paper/2607.21065/) (2026)
- [From Patches to Patients: A study of the tile-to-slide performance transferability in Digital Pathology](https://mlsanity.com/paper/2606.10778/) (2026)
- [Exploiting Spatial Structure for Transductive Few-Shot Classification of Whole-Slide Images](https://mlsanity.com/paper/2609.31040/) (2026)
- [PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology](https://mlsanity.com/paper/2610.11329/) (2026)
- [Scaling Multi-Teacher Distillation for Digital Pathology](https://mlsanity.com/paper/2610.12574/) (2026)

## Abstract (arXiv)

Recently, histopathology vision-language foundation models (VLMs) have gained popularity due to their enhanced performance and generalizability across different downstream tasks. However, most existing histopathology benchmarks are either unimodal or limited in terms of diversity of clinical tasks, organs, and acquisition instruments, as well as their partial availability to the public due to patient data privacy. As a consequence, there is a lack of comprehensive evaluation of existing histopathology VLMs on a unified benchmark setting that better reflects a wide range of clinical scenarios. To address this gap, we introduce HistoVL, a fully open-source comprehensive benchmark comprising images acquired using up to 11 various acquisition tools that are paired with specifically crafted captions by incorporating class names and diverse pathology descriptions. Our Histo-VL includes 26 organs, 31 cancer types, and a wide variety of tissue obtained from 14 heterogeneous patient cohorts, totaling more than 5 million patches obtained from over 41K WSIs viewed under various magnification levels. We systematically evaluate existing histopathology VLMs on Histo-VL to simulate diverse tasks performed by experts in real-world clinical scenarios. Our analysis reveals interesting findings, including large sensitivity of most existing histopathology VLMs to textual changes with a drop in balanced accuracy of up to 25% in tasks such as Metastasis detection, low robustness to adversarial attacks, as well as improper calibration of models evident through high ECE values and low model prediction confidence, all of which can affect their clinical implementation.