mlsanityData to 7 Oct 2026

WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image

arXiv 2412.02141 ↗First posted 3 Dec 2024cs.CV cs.CLcited by 37

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent advancements in computational pathology have produced patch-level Multi-modal Large Language Models (MLLMs), but these models are limited by their inability to analyze whole slide images (WSIs) comprehensively and their tendency to bypass crucial morphological features that pathologists rely on for diagnosis. To address these challenges, we first introduce WSI-Bench, a large-scale morphology-aware benchmark containing 180k VQA pairs from 9,850 WSIs across 30 cancer types, designed to evaluate MLLMs' understanding of morphological characteristics crucial for accurate diagnosis. Building upon this benchmark, we present WSI-LLaVA, a novel framework for gigapixel WSI understanding that employs a three-stage training approach: WSI-text alignment, feature space alignment, and task-specific instruction tuning. To better assess model performance in pathological contexts, we develop two specialized WSI metrics: WSI-Precision and WSI-Relevance. Experimental results demonstrate that WSI-LLaVA outperforms existing models across all capability dimensions, with a significant improvement in morphological analysis, establishing a clear correlation between morphological understanding and diagnostic accuracy.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 35 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

16 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding15 Oct

Other references · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2019Publicly Available Clinical BERT Embeddingscited by 995
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  5. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  6. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  7. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
  8. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
  9. 2024HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interactioncited by 43
  10. 2024WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answeringcited by 34
  11. 2023WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Imagescited by 31
  12. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
3 more
  1. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 19
  2. 2022PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domaincited by 12
  3. 2024PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathologycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 27.

37 papers.

Sort:

By area:

  1. 2025A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Taskscited by 15
  2. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  3. 2025OralGPT-Omni: A Versatile Dental Multimodal Large Language Modelcited by 10
  4. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  5. 2025WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image Analysiscited by 7
  6. 2025Navigating Gigapixel Pathology Images with Large Multimodal Modelscited by 7
  7. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
  8. 2026PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQAcited by 5
  9. 2025Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Controlcited by 3
  10. 2026Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoningcited by 3
  11. 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understandingcited by 3
  12. 2026Enhancing Pathological VLMs with Cross-scale Reasoningcited by 3
25 more
  1. 2026NuNext: Reframing Nucleus Detection as Next-Point Detectioncited by 2
  2. 2026CTIS-QA: Clinical Template-Informed Slide-level Question Answering for Pathologycited by 1
  3. 2026Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Imagescited by 1
  4. 2026PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMscited by 1
  5. 2026Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarkscited by 1
  6. 2026Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoningcited by 1
  7. 2026Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perceptioncited by 1
  8. 2026SlideBank: A Persistent Hierarchical Evidence Bank for Consistent Whole-Slide Reasoningcited by 1
  9. 2025The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Modelscited by 0
  10. 2025Contrastive Cross-Bag Augmentation for Multiple Instance Learning-based Whole Slide Image Classificationcited by 0
  11. 2026Automated Histopathology Report Generation via Pyramidal Feature Extraction and the UNI Foundation Modelcited by 0
  12. 2026PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretationcited by 0
  13. 2026Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Modelscited by 0
  14. 2026Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generationcited by 0
  15. 2026How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathologycited by 0
  16. 2026HERO: Hypothesis-Driven Evidence Retrieval from Omics for Multi-Task Breast Cancer Analysiscited by 0
  17. 2026LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Modelscited by 0
  18. 2026Pathologist Attention-Aligned Report Generation for Prostate Histopathologycited by 0
  19. 2026Towards Reliable Stain Transfer: An Iterative Data-Model Co-Optimization Framework Based on Multimodal Expert-Guided Assessmentcited by 0
  20. 2026Evaluating Agentic Bioinformatics through Function, Evidence, and Validationcited by 0
  21. 2026From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathologycited by 0
  22. 2026Spatial Message Passing in Language Space for Pathology Image Interpretationcited by 0
  23. 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generationcited by 0
  24. 2026SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Modelscited by 0
  25. 2026Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis
  2. 2024PathAlign: A vision-language model for whole slide images in histopathology
  3. 2025VLEER: Vision and Language Embeddings for Explainable Whole Slide Image Representation
  4. 2025PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
  5. 2025Lifelong Whole Slide Image Analysis: Online Vision-Language Adaptation and Past-to-Present Gradient Distillation
  6. 2026PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?
  7. 2025Efficient Whole Slide Pathology VQA via Token Compression
  8. 2026Exploiting Spatial Structure for Transductive Few-Shot Classification of Whole-Slide Images