mlsanityData to 14 Oct 2026

CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology

arXiv 2412.12077 ↗First posted 16 Dec 2024cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The emergence of large multimodal models (LMMs) has brought significant advancements to pathology. Previous research has primarily focused on separately training patch-level and whole-slide image (WSI)-level models, limiting the integration of learned knowledge across patches and WSIs, and resulting in redundant models. In this work, we introduce CPath-Omni, the first 15-billion-parameter LMM designed to unify both patch and WSI level image analysis, consolidating a variety of tasks at both levels, including classification, visual question answering, captioning, and visual referring prompting. Extensive experiments demonstrate that CPath-Omni achieves state-of-the-art (SOTA) performance across seven diverse tasks on 39 out of 42 datasets, outperforming or matching task-specific models trained for individual tasks. Additionally, we develop a specialized pathology CLIP-based visual processor for CPath-Omni, CPath-CLIP, which, for the first time, integrates different vision models and incorporates a large language model as a text encoder to build a more powerful CLIP model, which achieves SOTA performance on nine zero-shot and four few-shot datasets. Our findings highlight CPath-Omni's ability to unify diverse pathology tasks, demonstrating its potential to streamline and advance the field of foundation model in pathology.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 58 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

36 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May
  2. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology24 May

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  11. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  12. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 478
22 more
  1. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  2. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  3. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  4. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  5. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
  6. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  7. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  8. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 90
  9. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
  10. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  11. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 85
  12. 2019CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentationcited by 73
  13. 2021Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detectioncited by 68
  14. 2021A Petri Dish for Histopathology Image Analysiscited by 64
  15. 2021Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slidescited by 43
  16. 2024HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interactioncited by 43
  17. 2022WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinomacited by 36
  18. 2024WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answeringcited by 34
  19. 2023WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Imagescited by 31
  20. 2023OCELOT: Overlapped Cell on Tissue Dataset for Histopathologycited by 29
  21. 2023Domain generalization across tumor types, laboratories, and species -- insights from the 2022 edition of the Mitosis Domain Generalization Challengecited by 25
  22. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 18.

34 papers.

Sort:

By area:

  1. 2025Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasonercited by 40
  2. 2025CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logiccited by 29
  3. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 27
  4. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  5. 2026Atlas 2 -- Foundation models for clinical deploymentcited by 7
  6. 2026PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQAcited by 7
  7. 2025Efficient Whole Slide Pathology VQA via Token Compressioncited by 6
  8. 2025LoC-Path: Learning to Compress for Pathology Multimodal Large Language Modelscited by 6
  9. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  10. 2026Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoningcited by 5
  11. 2025Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoningcited by 4
  12. 2025PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathologycited by 4
22 more
  1. 2025MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retentioncited by 3
  2. 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understandingcited by 3
  3. 2025Foundation Models in Biomedical Imaging: Turning Hype into Realitycited by 2
  4. 2026NuNext: Reframing Nucleus Detection as Next-Point Detectioncited by 2
  5. 2026Do Pathology Vision-Language Models Truly See Pathology?cited by 2
  6. 2026PathSelect: Sequential Token Selection for Whole Slide Pathologycited by 2
  7. 2025Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generationcited by 1
  8. 2025Why Text Prevails: Vision May Undermine Multimodal Medical Decision Makingcited by 1
  9. 2026CTIS-QA: Clinical Template-Informed Slide-level Question Answering for Pathologycited by 1
  10. 2026Mixed Magnification Aggregation for Generalizable Region-Level Representations in Computational Pathologycited by 1
  11. 2026PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMscited by 1
  12. 2025VLM-based Prompts as the Optimal Assistant for Unpaired Histopathology Virtual Stainingcited by 0
  13. 2025ViDRiP-LLaVA: A Dataset and Benchmark for Diagnostic Reasoning from Pathology Videoscited by 0
  14. 2025GNN-ViTCap: GNN-Enhanced Multiple Instance Learning with Vision Transformers for Whole Slide Image Classification and Captioningcited by 0
  15. 2025Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtypingcited by 0
  16. 2026AINet: Anchor Instances Learning for Regional Heterogeneity in Whole Slide Imagecited by 0
  17. 2026PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretationcited by 0
  18. 2026How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathologycited by 0
  19. 2026Mitigating Batch Effects in Histopathology via Language-Mediated Robust Embedding Generationcited by 0
  20. 2026From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathologycited by 0
  21. 2026SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Modelscited by 0
  22. 2026Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PolyPath: Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation
  2. 2025LanGuSTE: Language-Guided Coarse-to-Fine Patch Selection for Efficient Whole Slide Image Analysis
  3. 2024SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding
  4. 2026PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?
  5. 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
  6. 2025MPath: Multimodal Pathology Report Generation from Whole Slide Images
  7. 2025Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation
  8. 2024Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis