mlsanityData to 14 Oct 2026

BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models

arXiv 2411.15232 ↗First posted 21 Nov 2024cs.CV cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent advancements in vision-language models (VLMs), such as CLIP, have demonstrated substantial success in self-supervised representation learning for vision tasks. However, effectively adapting VLMs to downstream applications remains challenging, as their accuracy often depends on time-intensive and expertise-demanding prompt engineering, while full model fine-tuning is costly. This is particularly true for biomedical images, which, unlike natural images, typically suffer from limited annotated datasets, unintuitive image contrasts, and nuanced visual features. Recent prompt learning techniques, such as Context Optimization (CoOp) intend to tackle these issues, but still fall short in generalizability. Meanwhile, explorations in prompt learning for biomedical image analysis are still highly limited. In this work, we propose BiomedCoOp, a novel prompt learning framework that enables efficient adaptation of BiomedCLIP for accurate and highly generalizable few-shot biomedical image classification. Our approach achieves effective prompt context learning by leveraging semantic consistency with average prompt ensembles from Large Language Models (LLMs) and knowledge distillation with a statistics-based prompt selection strategy. We conducted comprehensive validation of our proposed framework on 11 medical datasets across 9 modalities and 10 organs against existing state-of-the-art methods, demonstrating significant improvements in both accuracy and generalizability. The code is publicly available at https://github.com/HealthX-Lab/BiomedCoOp.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 46 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

32 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Aligning Medical Images with General Knowledge from Large Language Models31 Aug
  2. 2024XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization14 Mar

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  9. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 669
  10. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  11. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  12. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
18 more
  1. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  2. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  3. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  4. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  5. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 225
  6. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  7. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
  8. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  9. 2021COVID-19 Infection Localization and Severity Grading from Chest X-ray Imagescited by 46
  10. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 39
  11. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 38
  12. 2024MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentationcited by 37
  13. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 31
  14. 2023PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refinecited by 28
  15. 2018The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionscited by 26
  16. 2023Domain-Controlled Prompt Learningcited by 22
  17. 2024Prompt engineering paradigms for medical applications: scoping review and recommendations for better practicescited by 14
  18. 2024CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptationcited by 7
  2. 2025TextSAM-EUS: Text Prompt Learning for SAM to Accurately Segment Pancreatic Tumor in Endoscopic Ultrasoundcited by 5
  3. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classificationcited by 5
  4. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentationcited by 4
  5. 2025Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Modelscited by 3
  6. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Valuescited by 3
  7. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  8. 2026SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuningcited by 2
  9. 2025Interpreting Biomedical VLMs on High-Imbalance Out-of-Distributions: An Insight into BiomedCLIP on Radiologycited by 1
  10. 2026Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learningcited by 1
  11. 2026ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Modelscited by 1
  12. 2026CXR-LT 2026 Challenge: Projection-Aware Multi-Label and Zero-Shot Chest X-Ray Classificationcited by 1
16 more
  1. 2026Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMscited by 1
  2. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQAcited by 1
  3. 2025Prompt Mechanisms in Medical Imaging: A Comprehensive Surveycited by 0
  4. 2025BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Modelscited by 0
  5. 2025Cross-Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptationcited by 0
  6. 2026XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Modelscited by 0
  7. 2026K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imagingcited by 0
  8. 2026Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classificationcited by 0
  9. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptationcited by 0
  10. 2026Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuningcited by 0
  11. 2026Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Modelscited by 0
  12. 2026Evo-RAD: Navigating Rare Retinal Disease Diagnosis via Self-Evolving Agentic Retrievalcited by 0
  13. 2026BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogrammingcited by 0
  14. 2026TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Modelscited by 0
  15. 2026Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shiftcited by 0
  16. 2026Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models
  2. 2025Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models
  3. 2025vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs
  4. 2026BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
  5. 2025Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
  6. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
  7. 2025Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks
  8. 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations