# MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding arXiv: https://arxiv.org/abs/2603.23067 ยท HTML: https://mlsanity.com/paper/2603.23067/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 35 of its 88 references that are in the map. - First posted on arXiv 24 Mar 2026. ## Placed in - Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/) ## Builds on (most cited first, 30 of 35) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023) - [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023) - [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023) - [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019) - [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024) - [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024) - [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018) - [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022) - [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023) - [Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology](https://mlsanity.com/paper/2408.00738/) (2024) - [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022) - [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023) - [BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images](https://mlsanity.com/paper/2111.04740/) (2021) - [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023) - [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023) - [PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology](https://mlsanity.com/paper/2405.10254/) (2024) - [Phikon-v2, A large and public feature extractor for biomarker prediction](https://mlsanity.com/paper/2409.09173/) (2024) - [PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology](https://mlsanity.com/paper/2305.15072/) (2023) - [SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding](https://mlsanity.com/paper/2410.11761/) (2024) - [Micro-Net: A unified model for segmentation of various objects in microscopy images](https://mlsanity.com/paper/1804.08145/) (2018) - [HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction](https://mlsanity.com/paper/2403.05396/) (2024) - [The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification](https://mlsanity.com/paper/2305.17891/) (2023) - [RudolfV: A Foundation Model by Pathologists for Pathologists](https://mlsanity.com/paper/2401.04079/) (2024) - [Robust and Efficient Medical Imaging with Self-Supervision](https://mlsanity.com/paper/2205.09723/) (2022) - [WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image](https://mlsanity.com/paper/2412.02141/) (2024) - [WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering](https://mlsanity.com/paper/2407.05603/) (2024) - [CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology](https://mlsanity.com/paper/2412.12077/) (2024) - [Nuclei instance segmentation and classification in histopathology images with StarDist](https://mlsanity.com/paper/2203.02284/) (2022) ## Built on by (most cited first, 3 of 3) - [Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks](https://mlsanity.com/paper/2607.12278/) (2026) - [SlideBank: A Persistent Hierarchical Evidence Bank for Consistent Whole-Slide Reasoning](https://mlsanity.com/paper/2609.00342/) (2026) - [How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology](https://mlsanity.com/paper/2606.12407/) (2026) ## Abstract (arXiv) Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce MLLM-HWSI, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight textitCell-Cell Attention Fusion (CCAF) transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: hrefhttps://github.com/BasitAlawode/HWSI-MLLMGitHub.