mlsanityData to 14 Oct 2026

Effortless Vision-Language Model Specialization in Histopathology without Annotation

arXiv 2508.07835 ↗First posted 11 Aug 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent advances in Vision-Language Models (VLMs) in histopathology, such as CONCH and QuiltNet, have demonstrated impressive zero-shot classification capabilities across various tasks. However, their general-purpose design may lead to suboptimal performance in specific downstream applications. While supervised fine-tuning methods address this issue, they require manually labeled samples for adaptation. This paper investigates annotation-free adaptation of VLMs through continued pretraining on domain- and task-relevant image-caption pairs extracted from existing databases. Our experiments on two VLMs, CONCH and QuiltNet, across three downstream tasks reveal that these pairs substantially enhance both zero-shot and few-shot performance. Notably, with larger training sizes, continued pretraining matches the performance of few-shot methods while eliminating manual labeling. Its effectiveness, task-agnostic design, and annotation-free workflow make it a promising pathway for adapting VLMs to new histopathology tasks. Code is available at https://github.com/DeepMicroscopy/Annotation-free-VLM-specialization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 21 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

18 of 21 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  5. 2025Gemma 3 Technical Reportcited by 1,824
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  10. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  11. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  12. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
6 more
  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 119
  2. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
  3. 2021Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detectioncited by 68
  4. 2021A Petri Dish for Histopathology Image Analysiscited by 64
  5. 2022PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domaincited by 12
  6. 2024Model-based Cleaning of the QUILT-1M Pathology Dataset for Text-Conditional Image Synthesiscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at once
  2. 2025Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
  3. 2024MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological images
  4. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design
  5. 2025Slide-Level Prompt Learning with Vision Language Models for Few-Shot Multiple Instance Learning in Histopathology
  6. 2024CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment
  7. 2026Histopath-C: Towards Realistic Domain Shifts for Histopathology Vision-Language Adaptation
  8. 2026Initialization matters in few-shot adaptation of vision-language models for histopathological image classification