mlsanityData to 14 Oct 2026

Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosis

arXiv 2412.13126 ↗First posted 17 Dec 2024eess.IV cs.CVcited by 21

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language foundation models have shown great promise in computational pathology but remain primarily data-driven, lacking explicit integration of medical knowledge. We introduce KEEP (KnowledgE-Enhanced Pathology), a foundation model that systematically incorporates disease knowledge into pretraining for cancer diagnosis. KEEP leverages a comprehensive disease knowledge graph encompassing 11,454 diseases and 139,143 attributes to reorganize millions of pathology image-text pairs into 143,000 semantically structured groups aligned with disease ontology hierarchies. This knowledge-enhanced pretraining aligns visual and textual representations within hierarchical semantic spaces, enabling deeper understanding of disease relationships and morphological patterns. Across 18 public benchmarks (over 14,000 whole-slide images) and 4 institutional rare cancer datasets (926 cases), KEEP consistently outperformed existing foundation models, showing substantial gains for rare subtypes. These results establish knowledge-enhanced vision-language modeling as a powerful paradigm for advancing computational pathology.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 65 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

29 of 65 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  5. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  6. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
  7. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 555
  8. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  9. 2023Segment Anything in Medical Imagescited by 227
  10. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  11. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  12. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
17 more
  1. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
  2. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134
  3. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  4. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 111
  5. 2023Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Imagescited by 91
  6. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
  7. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 85
  8. 2024A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Modelcited by 59
  9. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  10. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  11. 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
  12. 2023Histopathology Whole Slide Image Analysis with Heterogeneous Graph Representation Learningcited by 47
  13. 2019Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Imagescited by 39
  14. 2022WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinomacited by 36
  15. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  16. 2023Adaptive Sparse Pairwise Loss for Object Re-Identificationcited by 20
  17. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  2. 2025A Survey of Pathology Foundation Model: Progress and Future Directionscited by 16
  3. 2025PixCell: A generative foundation model for digital histopathology imagescited by 11
  4. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  5. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 8
  6. 2025TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learningcited by 6
  7. 2025Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignmentcited by 4
  8. 2025Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtypingcited by 3
  9. 2025DepViT-CAD: Deployable Vision Transformer-Based Cancer Diagnosis in Histopathologycited by 1
  10. 2025Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generationcited by 1
  11. 2025Comparing Computational Pathology Foundation Models using Representational Similarity Analysiscited by 0
  12. 2025Zero-shot segmentation of skin tumors in whole-slide images with vision-language foundation modelscited by 0
9 more
  1. 2026PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision-Language Pretrainingcited by 0
  2. 2026GRMLR: Knowledge-Enhanced Small-Data Learning for Deep-Sea Cold Seep Stage Inferencecited by 0
  3. 2026VBFDD-Agent for Electric Vehicle Battery Fault Detection and Diagnosis: Descriptive Text Modeling of Battery Digital Signalscited by 0
  4. 2026One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensemblingcited by 0
  5. 2026APRIL-MedSeg: A Modular Medical Image Segmentation Toolbox Embracing Modern Paradigmscited by 0
  6. 2026Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathologycited by 0
  7. 2026FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathologycited by 0
  8. 2026Do Center Biases Propagate? Robustness of Pathology Foundation Models in Whole-Slide Image Classificationcited by 0
  9. 2026Refining Cytology Predictions with Conditional Random Fieldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology
  2. 2025DiagR1: A Vision-Language Model Trained via Reinforcement Learning for Digestive Pathology Diagnosis
  3. 2026PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs
  4. 2024Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation
  5. 2026PathReasoner-R1: Instilling Structured Reasoning into Pathology Vision-Language Model via Knowledge-Guided Policy Optimization
  6. 2026See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology
  7. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework
  8. 2024Benchmarking foundation models as feature extractors for weakly-supervised computational pathology