PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue
Not on the map. Why?
Abstract (arXiv)
Recent rapid progress in the field of computational pathology has been enabled by foundation models. These models are beginning to move beyond encoding image patches towards whole-slide understanding but their clinical utility remains limited. In this work, we present PRISM2, a multimodal slide-level foundation model trained on data from 700,000 diagnostic specimen-report pairs, the largest vision (2.3 million whole slide images) and language (14M question-answer pairs) histopathology dataset to date. By learning through clinical-dialogue supervision, PRISM2 aligns histomorphologic features with the language of diagnostic reasoning, producing slide-level representations that support both direct diagnostic question-answering and transferable embeddings for downstream tasks. Without additional training, PRISM2 matches or exceeds the cancer-detection performance of clinical-grade products. This is observed without loss of generality on other tasks, where PRISM2 achieves top performance. Finally, using survival prediction as the example, we show that task-specific finetuning with a large dataset can outperform task-specific models, further improving performance. These results demonstrate how language-supervised pretraining provides a scalable, clinically grounded signal for learning generalizable pathology representations, bridging human diagnostic reasoning and foundation-model performance.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 54 references that are in the map. First posted on arXiv 16 Jun 2025.
Builds on
27 of 54 references.
References · 27
- 2023Visual Instruction Tuningcited by 7,792
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2024GPT-4o System Cardcited by 4,467
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
- 2021Scaling Vision Transformerscited by 993
15 more
- 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
- 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
- 2024Multimodal Whole Slide Foundation Model for Pathologycited by 156
- 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
- 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
- 2023Artificial Intelligence for Digital and Computational Pathologycited by 106
- 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
- 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
- 2024A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Modelcited by 59
- 2025Molecular-driven Foundation Model for Oncologic Pathologycited by 52
- 2024Transcriptomics-guided Slide Representation Learning in Computational Pathologycited by 50
- 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 29
- 2024Multistain Pretraining for Slide Representation Learning in Pathologycited by 27
- 2024Unsupervised Foundation Model-Agnostic Slide-Level Representation Learningcited by 15
- 2024Screen Them All: High-Throughput Pan-Cancer Genetic and Phenotypic Biomarker Screening from H&E Whole Slide Imagescited by 8
Built on by
Citing papers per year: 2025: 5; 2026: 3.
8 papers.
- 2025PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoningcited by 18
- 2025Navigating Gigapixel Pathology Images with Large Multimodal Modelscited by 7
- 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
- 2025MedFoundationHub: A Lightweight and Secure Toolkit for Deploying Medical Vision Language Foundation Modelscited by 0
- 2025Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignmentcited by 0
- 2026DALPHIN: Benchmarking Digital Pathology AI Copilots Against Pathologists on an Open Multicentric Datasetcited by 0
- 2026PathoSage: Towards Multi-Source Evidence Adjudication in Pathology via Experience-Aware Agentic Workflowcited by 0
- 2026Mitigating Batch Effects in Histopathology via Language-Mediated Robust Embedding Generationcited by 0
- 2026Mitigating Batch Effects in Histopathology via Language-Mediated Robust Embedding Generation27 Jun 2026
- 2026PathoSage: Towards Multi-Source Evidence Adjudication in Pathology via Experience-Aware Agentic Workflow18 May 2026
- 2026DALPHIN: Benchmarking Digital Pathology AI Copilots Against Pathologists on an Open Multicentric Dataset5 May 2026
- 2025Navigating Gigapixel Pathology Images with Large Multimodal Models24 Nov 2025
- 2025PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning21 Nov 2025
- 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilots20 Nov 2025
- 2025Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment27 Oct 2025
- 2025MedFoundationHub: A Lightweight and Secure Toolkit for Deploying Medical Vision Language Foundation Models28 Aug 2025