# WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image arXiv: https://arxiv.org/abs/2412.02141 ยท HTML: https://mlsanity.com/paper/2412.02141/ ## Key facts - Cited by 40 papers in the map (data through 14 Oct 2026). - Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 16 of its 35 references that are in the map. - First posted on arXiv 3 Dec 2024. ## Placed in - Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/) ## Builds on (most cited first, 16 of 16) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019) - [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020) - [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023) - [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023) - [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023) - [PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology](https://mlsanity.com/paper/2405.10254/) (2024) - [SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding](https://mlsanity.com/paper/2410.11761/) (2024) - [HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction](https://mlsanity.com/paper/2403.05396/) (2024) - [WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering](https://mlsanity.com/paper/2407.05603/) (2024) - [WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images](https://mlsanity.com/paper/2311.16480/) (2023) - [A Foundational Multimodal Vision Language AI Assistant for Human Pathology](https://mlsanity.com/paper/2312.07814/) (2023) - [MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation](https://mlsanity.com/paper/2310.14088/) (2023) - [PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domain](https://mlsanity.com/paper/2205.06885/) (2022) - [PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology](https://mlsanity.com/paper/2408.07037/) (2024) ## Built on by (most cited first, 30 of 40) - [A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Tasks](https://mlsanity.com/paper/2501.15724/) (2025) - [EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis](https://mlsanity.com/paper/2505.23601/) (2025) - [OralGPT-Omni: A Versatile Dental Multimodal Large Language Model](https://mlsanity.com/paper/2511.22055/) (2025) - [Multi-Modal Foundation Models for Computational Pathology: A Survey](https://mlsanity.com/paper/2503.09091/) (2025) - [WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image Analysis](https://mlsanity.com/paper/2507.14680/) (2025) - [Navigating Gigapixel Pathology Images with Large Multimodal Models](https://mlsanity.com/paper/2511.19652/) (2025) - [PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA](https://mlsanity.com/paper/2605.23559/) (2026) - [LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models](https://mlsanity.com/paper/2512.05391/) (2025) - [Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning](https://mlsanity.com/paper/2603.00667/) (2026) - [Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Control](https://mlsanity.com/paper/2512.21058/) (2025) - [MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding](https://mlsanity.com/paper/2603.23067/) (2026) - [Enhancing Pathological VLMs with Cross-scale Reasoning](https://mlsanity.com/paper/2606.17412/) (2026) - [NuNext: Reframing Nucleus Detection as Next-Point Detection](https://mlsanity.com/paper/2603.07098/) (2026) - [Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoning](https://mlsanity.com/paper/2608.05757/) (2026) - [CTIS-QA: Clinical Template-Informed Slide-level Question Answering for Pathology](https://mlsanity.com/paper/2601.01769/) (2026) - [Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images](https://mlsanity.com/paper/2602.19424/) (2026) - [PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs](https://mlsanity.com/paper/2603.09943/) (2026) - [Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks](https://mlsanity.com/paper/2607.12278/) (2026) - [Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception](https://mlsanity.com/paper/2608.08648/) (2026) - [SlideBank: A Persistent Hierarchical Evidence Bank for Consistent Whole-Slide Reasoning](https://mlsanity.com/paper/2609.00342/) (2026) - [The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Models](https://mlsanity.com/paper/2505.24141/) (2025) - [Contrastive Cross-Bag Augmentation for Multiple Instance Learning-based Whole Slide Image Classification](https://mlsanity.com/paper/2508.03081/) (2025) - [Automated Histopathology Report Generation via Pyramidal Feature Extraction and the UNI Foundation Model](https://mlsanity.com/paper/2602.16422/) (2026) - [PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretation](https://mlsanity.com/paper/2604.17570/) (2026) - [Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models](https://mlsanity.com/paper/2605.24799/) (2026) - [Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation](https://mlsanity.com/paper/2605.30716/) (2026) - [How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology](https://mlsanity.com/paper/2606.12407/) (2026) - [HERO: Hypothesis-Driven Evidence Retrieval from Omics for Multi-Task Breast Cancer Analysis](https://mlsanity.com/paper/2606.21174/) (2026) - [LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models](https://mlsanity.com/paper/2607.11257/) (2026) - [Pathologist Attention-Aligned Report Generation for Prostate Histopathology](https://mlsanity.com/paper/2607.19624/) (2026) ## Abstract (arXiv) Recent advancements in computational pathology have produced patch-level Multi-modal Large Language Models (MLLMs), but these models are limited by their inability to analyze whole slide images (WSIs) comprehensively and their tendency to bypass crucial morphological features that pathologists rely on for diagnosis. To address these challenges, we first introduce WSI-Bench, a large-scale morphology-aware benchmark containing 180k VQA pairs from 9,850 WSIs across 30 cancer types, designed to evaluate MLLMs' understanding of morphological characteristics crucial for accurate diagnosis. Building upon this benchmark, we present WSI-LLaVA, a novel framework for gigapixel WSI understanding that employs a three-stage training approach: WSI-text alignment, feature space alignment, and task-specific instruction tuning. To better assess model performance in pathological contexts, we develop two specialized WSI metrics: WSI-Precision and WSI-Relevance. Experimental results demonstrate that WSI-LLaVA outperforms existing models across all capability dimensions, with a significant improvement in morphological analysis, establishing a clear correlation between morphological understanding and diagnostic accuracy.