mlsanityData to 7 Oct 2026

CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment

arXiv 2406.05205 ↗First posted 7 Jun 2024cs.CV cs.CL cs.LGcited by 20

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

This paper proposes Comprehensive Pathology Language Image Pre-training (CPLIP), a new unsupervised technique designed to enhance the alignment of images and text in histopathology for tasks such as classification and segmentation. This methodology enriches vision-language models by leveraging extensive data without needing ground truth annotations. CPLIP involves constructing a pathology-specific dictionary, generating textual descriptions for images using language models, and retrieving relevant images for each text snippet via a pre-trained model. The model is then fine-tuned using a many-to-many contrastive learning method to align complex interrelated concepts across both modalities. Evaluated across multiple histopathology tasks, CPLIP shows notable improvements in zero-shot learning scenarios, outperforming existing methods in both interpretability and robustness and setting a higher benchmark for the application of vision-language models in the field. To encourage further research and replication, the code for CPLIP is available on GitHub at https://cplip.github.io/

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 49 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

27 of 49 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  7. 2019Publicly Available Clinical BERT Embeddingscited by 995
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  10. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  12. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
15 more
  1. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  2. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  3. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 254
  4. 2019Deep neural network models for computational histopathology: A surveycited by 165
  5. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  6. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  7. 2020Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotationscited by 89
  8. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  9. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  10. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  11. 2023CiteTracker: Correlating Image and Text for Visual Trackingcited by 36
  12. 2023A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutionscited by 34
  13. 2022Handcrafted Histological Transformer (H2T): Unsupervised Representation of Whole Slide Imagescited by 32
  14. 2022WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinomacited by 32
  15. 2021Self-learning for weakly supervised Gleason grading of local patternscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 3.

20 papers.

Sort:

By area:

  1. 2024Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathologycited by 14
  2. 2025Interactive Medical Image Analysis with Concept-based Similarity Reasoningcited by 11
  3. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  4. 2025GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathologycited by 8
  5. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generationcited by 7
  6. 2025Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representationcited by 6
  7. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  8. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  9. 2025Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challengescited by 5
  10. 2025Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignmentcited by 4
  11. 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understandingcited by 3
  12. 2025VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicinecited by 2
8 more
  1. 2025Dynamic Residual Encoding with Slide-Level Contrastive Learning for End-to-End Whole Slide Image Representationcited by 2
  2. 2025MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Imagescited by 2
  3. 2025PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathologycited by 1
  4. 2025CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selectioncited by 1
  5. 2025Dynamic Rank Adaptation for Vision-Language Modelscited by 0
  6. 2025MV-MLM: Bridging Multi-View Mammography and Language for Breast Cancer Diagnosis and Risk Predictioncited by 0
  7. 2026MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestrationcited by 0
  8. 2026Towards reliable multimodal disaster severity assessment through preference optimization and explainable vision-language reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Effortless Vision-Language Model Specialization in Histopathology without Annotation
  2. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  3. 2024PathAlign: A vision-language model for whole slide images in histopathology
  4. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
  5. 2025Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images
  6. 2024MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological images
  7. 2025Slide-Level Prompt Learning with Vision Language Models for Few-Shot Multiple Instance Learning in Histopathology
  8. 2026HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction