# PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

arXiv: https://arxiv.org/abs/2405.10254 · HTML: https://mlsanity.com/paper/2405.10254/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Virchow: A Million-Slide Digital Pathology Foundation Model](https://mlsanity.com/paper/2309.07778/) (2023)
- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [Towards a Visual-Language Foundation Model for Computational Pathology](https://mlsanity.com/paper/2307.12914/) (2023)
- [RudolfV: A Foundation Model by Pathologists for Pathologists](https://mlsanity.com/paper/2401.04079/) (2024)
- [Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images](https://mlsanity.com/paper/2310.07033/) (2023)
- [Inference of captions from histopathological patches](https://mlsanity.com/paper/2202.03432/) (2022)
- [When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology](https://mlsanity.com/paper/2312.03558/) (2023)
- [What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning](https://mlsanity.com/paper/2310.20607/) (2023)
- [The Importance of Downstream Networks in Digital Pathology Foundation Models](https://mlsanity.com/paper/2311.17804/) (2023)

## Built on by (most cited first, 30 of 85)

- [Multimodal Whole Slide Foundation Model for Pathology](https://mlsanity.com/paper/2411.19666/) (2024)
- [WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image](https://mlsanity.com/paper/2412.02141/) (2024)
- [CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology](https://mlsanity.com/paper/2412.12077/) (2024)
- [PathAlign: A vision-language model for whole slide images in histopathology](https://mlsanity.com/paper/2406.19578/) (2024)
- [CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic](https://mlsanity.com/paper/2505.20510/) (2025)
- [Do Multiple Instance Learning Models Transfer?](https://mlsanity.com/paper/2506.09022/) (2025)
- [Multistain Pretraining for Slide Representation Learning in Pathology](https://mlsanity.com/paper/2408.02859/) (2024)
- [Evidence-based diagnostic reasoning with multi-agent copilot for human pathology](https://mlsanity.com/paper/2506.20964/) (2025)
- [Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosis](https://mlsanity.com/paper/2412.13126/) (2024)
- [A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Tasks](https://mlsanity.com/paper/2501.15724/) (2025)
- [A Survey of Pathology Foundation Model: Progress and Future Directions](https://mlsanity.com/paper/2504.04045/) (2025)
- [Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning](https://mlsanity.com/paper/2411.13623/) (2024)
- [Atlas: A Novel Pathology Foundation Model by Mayo Clinic, Charit'e, and Aignostics](https://mlsanity.com/paper/2501.05409/) (2025)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [Multi-Modal Foundation Models for Computational Pathology: A Survey](https://mlsanity.com/paper/2503.09091/) (2025)
- [PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue](https://mlsanity.com/paper/2506.13063/) (2025)
- [A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics](https://mlsanity.com/paper/2508.01490/) (2025)
- [Pathology Foundation Models](https://mlsanity.com/paper/2407.21317/) (2024)
- [WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image Analysis](https://mlsanity.com/paper/2507.14680/) (2025)
- [Navigating Gigapixel Pathology Images with Large Multimodal Models](https://mlsanity.com/paper/2511.19652/) (2025)
- [Atlas 2 -- Foundation models for clinical deployment](https://mlsanity.com/paper/2601.05148/) (2026)
- [Pathology Report Generation and Multimodal Representation Learning for Cutaneous Melanocytic Lesions](https://mlsanity.com/paper/2502.19293/) (2025)
- [From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research](https://mlsanity.com/paper/2507.09028/) (2025)
- [LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models](https://mlsanity.com/paper/2512.05391/) (2025)
- [TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning](https://mlsanity.com/paper/2512.21331/) (2025)
- [CARE: A Molecular-Guided Foundation Model with Adaptive Region Modeling for Whole Slide Image Analysis](https://mlsanity.com/paper/2602.21637/) (2026)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [Fusion of Multi-scale Heterogeneous Pathology Foundation Models for Whole Slide Image Analysis](https://mlsanity.com/paper/2510.27237/) (2025)
- [PolyPath: Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation](https://mlsanity.com/paper/2502.10536/) (2025)
- [QCAgent: An agentic framework for quality-controllable pathology report generation from whole slide image](https://mlsanity.com/paper/2603.01647/) (2026)

## Abstract (arXiv)

Foundation models in computational pathology promise to unlock the development of new clinical decision support systems and models for precision medicine. However, there is a mismatch between most clinical analysis, which is defined at the level of one or more whole slide images, and foundation models to date, which process the thousands of image tiles contained in a whole slide image separately. The requirement to train a network to aggregate information across a large number of tiles in multiple whole slide images limits these models' impact. In this work, we present a slide-level foundation model for H&E-stained histopathology, PRISM, that builds on Virchow tile embeddings and leverages clinical report text for pre-training. Using the tile embeddings, PRISM produces slide-level embeddings with the ability to generate clinical reports, resulting in several modes of use. Using text prompts, PRISM achieves zero-shot cancer detection and sub-typing performance approaching and surpassing that of a supervised aggregator model. Using the slide embeddings with linear classifiers, PRISM surpasses supervised aggregator models. Furthermore, we demonstrate that fine-tuning of the PRISM slide encoder yields label-efficient training for biomarker prediction, a task that typically suffers from low availability of training data; an aggregator initialized with PRISM and trained on as little as 10% of the training data can outperform a supervised baseline that uses all of the data.