# CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology

arXiv: https://arxiv.org/abs/2412.12077 · HTML: https://mlsanity.com/paper/2412.12077/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 58 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models](https://mlsanity.com/paper/2310.08825/) (2023)
- [Virchow: A Million-Slide Digital Pathology Foundation Model](https://mlsanity.com/paper/2309.07778/) (2023)
- [Lung and Colon Cancer Histopathological Image Dataset (LC25000)](https://mlsanity.com/paper/1912.12142/) (2019)
- [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023)
- [PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology](https://mlsanity.com/paper/2405.10254/) (2024)
- [CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentation](https://mlsanity.com/paper/1908.10555/) (2019)
- [Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detection](https://mlsanity.com/paper/2105.10490/) (2021)
- [A Petri Dish for Histopathology Image Analysis](https://mlsanity.com/paper/2101.12355/) (2021)
- [PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology](https://mlsanity.com/paper/2305.15072/) (2023)
- [Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slides](https://mlsanity.com/paper/2112.02222/) (2021)
- [HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction](https://mlsanity.com/paper/2403.05396/) (2024)

## Built on by (most cited first, 30 of 34)

- [Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner](https://mlsanity.com/paper/2505.11404/) (2025)
- [CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic](https://mlsanity.com/paper/2505.20510/) (2025)
- [Evidence-based diagnostic reasoning with multi-agent copilot for human pathology](https://mlsanity.com/paper/2506.20964/) (2025)
- [Multi-Modal Foundation Models for Computational Pathology: A Survey](https://mlsanity.com/paper/2503.09091/) (2025)
- [Atlas 2 -- Foundation models for clinical deployment](https://mlsanity.com/paper/2601.05148/) (2026)
- [PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA](https://mlsanity.com/paper/2605.23559/) (2026)
- [Efficient Whole Slide Pathology VQA via Token Compression](https://mlsanity.com/paper/2507.14497/) (2025)
- [LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models](https://mlsanity.com/paper/2512.05391/) (2025)
- [Multimodal Model for Computational Pathology:Representation Learning and Image Compression](https://mlsanity.com/paper/2603.18660/) (2026)
- [Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning](https://mlsanity.com/paper/2603.00667/) (2026)
- [Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoning](https://mlsanity.com/paper/2505.15687/) (2025)
- [PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology](https://mlsanity.com/paper/2512.17621/) (2025)
- [MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention](https://mlsanity.com/paper/2503.00374/) (2025)
- [MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding](https://mlsanity.com/paper/2603.23067/) (2026)
- [Foundation Models in Biomedical Imaging: Turning Hype into Reality](https://mlsanity.com/paper/2512.15808/) (2025)
- [NuNext: Reframing Nucleus Detection as Next-Point Detection](https://mlsanity.com/paper/2603.07098/) (2026)
- [Do Pathology Vision-Language Models Truly See Pathology?](https://mlsanity.com/paper/2607.21065/) (2026)
- [PathSelect: Sequential Token Selection for Whole Slide Pathology](https://mlsanity.com/paper/2607.23631/) (2026)
- [Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generation](https://mlsanity.com/paper/2512.03445/) (2025)
- [Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making](https://mlsanity.com/paper/2512.13747/) (2025)
- [CTIS-QA: Clinical Template-Informed Slide-level Question Answering for Pathology](https://mlsanity.com/paper/2601.01769/) (2026)
- [Mixed Magnification Aggregation for Generalizable Region-Level Representations in Computational Pathology](https://mlsanity.com/paper/2602.22176/) (2026)
- [PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs](https://mlsanity.com/paper/2603.09943/) (2026)
- [VLM-based Prompts as the Optimal Assistant for Unpaired Histopathology Virtual Staining](https://mlsanity.com/paper/2504.15545/) (2025)
- [ViDRiP-LLaVA: A Dataset and Benchmark for Diagnostic Reasoning from Pathology Videos](https://mlsanity.com/paper/2505.04192/) (2025)
- [GNN-ViTCap: GNN-Enhanced Multiple Instance Learning with Vision Transformers for Whole Slide Image Classification and Captioning](https://mlsanity.com/paper/2507.07006/) (2025)
- [Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtyping](https://mlsanity.com/paper/2508.15904/) (2025)
- [AINet: Anchor Instances Learning for Regional Heterogeneity in Whole Slide Image](https://mlsanity.com/paper/2602.20187/) (2026)
- [PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretation](https://mlsanity.com/paper/2604.17570/) (2026)
- [How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology](https://mlsanity.com/paper/2606.12407/) (2026)

## Abstract (arXiv)

The emergence of large multimodal models (LMMs) has brought significant advancements to pathology. Previous research has primarily focused on separately training patch-level and whole-slide image (WSI)-level models, limiting the integration of learned knowledge across patches and WSIs, and resulting in redundant models. In this work, we introduce CPath-Omni, the first 15-billion-parameter LMM designed to unify both patch and WSI level image analysis, consolidating a variety of tasks at both levels, including classification, visual question answering, captioning, and visual referring prompting. Extensive experiments demonstrate that CPath-Omni achieves state-of-the-art (SOTA) performance across seven diverse tasks on 39 out of 42 datasets, outperforming or matching task-specific models trained for individual tasks. Additionally, we develop a specialized pathology CLIP-based visual processor for CPath-Omni, CPath-CLIP, which, for the first time, integrates different vision models and incorporates a large language model as a text encoder to build a more powerful CLIP model, which achieves SOTA performance on nine zero-shot and four few-shot datasets. Our findings highlight CPath-Omni's ability to unify diverse pathology tasks, demonstrating its potential to streamline and advance the field of foundation model in pathology.