mlsanityData to 14 Oct 2026

Knowledge-enhanced Visual-Language Pretraining for Computational Pathology

arXiv 2404.09942 ↗First posted 15 Apr 2024cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources, along with the domain-specific knowledge in pathology. Specifically, we make the following contributions: (i) We curate a pathology knowledge tree that consists of 50,470 informative attributes for 4,718 diseases requiring pathology diagnosis from 32 human tissues. To our knowledge, this is the first comprehensive structured pathology knowledge base; (ii) We develop a knowledge-enhanced visual-language pretraining approach, where we first project pathology-specific knowledge into latent embedding space via a language model, and use it to guide the visual representation learning; (iii) We conduct thorough experiments to validate the effectiveness of our proposed components, demonstrating significant performance improvement on various downstream tasks, including cross-modal retrieval, zero-shot classification on pathology patches, and zero-shot tumor subtyping on whole slide images (WSIs).

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 44 of its 65 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

44 of 65 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  11. 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 786
  12. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
32 more
  1. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  3. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  4. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  5. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  6. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  7. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  8. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  9. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  10. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  11. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
  12. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  13. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134
  14. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  15. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  16. 2023Interventional Bag Multi-Instance Learning On Whole-Slide Pathological Imagescited by 91
  17. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
  18. 2020Hard negative examples are hard, but usefulcited by 82
  19. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  20. 2021Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detectioncited by 68
  21. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  22. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  23. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  24. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  25. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  26. 2023Histopathology Whole Slide Image Analysis with Heterogeneous Graph Representation Learningcited by 47
  27. 2019Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Imagescited by 39
  28. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  29. 2022WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinomacited by 36
  30. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  31. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  32. 2023Adaptive Sparse Pairwise Loss for Object Re-Identificationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  2. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  3. 2024Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosiscited by 21
  4. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  5. 2025A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Taskscited by 16
  6. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  7. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  8. 2025GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathologycited by 8
  9. 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understandingcited by 3
  10. 2026UniSemAlign: Text-Prototype Alignment with a Foundation Encoder for Semi-Supervised Histopathology Segmentationcited by 2
  11. 2025Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generationcited by 1
  12. 2025Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtypingcited by 0
2 more
  1. 2026PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision-Language Pretrainingcited by 0
  2. 2026From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathologycited by 0