mlsanityData to 14 Oct 2026

UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts

arXiv 2312.11171 ↗First posted 18 Dec 2023cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical vision-language pre-training (Med-VLP) models have recently accelerated the fast-growing medical diagnostics application. However, most Med-VLP models learn task-specific representations independently from scratch, thereby leading to great inflexibility when they work across multiple fine-tuning tasks. In this work, we propose UniDCP, a Unified medical vision-language model with Dynamic Cross-modal learnable Prompts, which can be plastically applied to multiple medical vision-language tasks. Specifically, we explicitly construct a unified framework to harmonize diverse inputs from multiple pretraining tasks by leveraging cross-modal prompts for unification, which accordingly can accommodate heterogeneous medical fine-tuning tasks. Furthermore, we conceive a dynamic cross-modal prompt optimizing strategy that optimizes the prompts within the shareable space for implicitly processing the shareable clinic knowledge. UniDCP is the first Med-VLP model capable of performing all 8 medical uni-modal and cross-modal tasks over 14 corresponding datasets, consistently yielding superior results over diverse state-of-the-art methods.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 71 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

51 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training13 May

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2022Visual Prompt Tuningcited by 1,681
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  12. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
38 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  6. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  7. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  8. 2016Self-critical Sequence Training for Image Captioningcited by 547
  9. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  11. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  12. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  13. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  14. 2018Bilinear Attention Networkscited by 280
  15. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  16. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  17. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
  18. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  19. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  20. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  21. 2022Unified Vision and Language Prompt Learningcited by 148
  22. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  23. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  24. 2020Text-to-Text Pre-Training for Data-to-Text Taskscited by 104
  25. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  26. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  27. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  28. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  29. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  30. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 73
  31. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  32. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  33. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  34. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 52
  35. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 36
  36. 2020An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Miningcited by 35
  37. 2020Results of the seventh edition of the BioASQ Challengecited by 28
  38. 2020MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classificationcited by 9

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  2. 2024Mitigating Multilingual Hallucination in Large Vision-Language Modelscited by 3
  3. 2024Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?cited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
  2. 2024Uni-Mlip: Unified Self-supervision for Medical Vision Language Pre-training
  3. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
  4. 2026SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
  5. 2026MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration
  6. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE
  7. 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations
  8. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?