mlsanityData to 14 Oct 2026

Deeply Coupled Cross-Modal Prompt Learning

arXiv 2305.17903 ↗First posted 29 May 2023cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advancements in multimodal foundation models (e.g., CLIP) have excelled in zero-shot generalization. Prompt tuning involved in the knowledge transfer from foundation models to downstream tasks has gained significant attention recently. Existing prompt-tuning methods in cross-modal learning, however, either solely focus on language branch, or learn vision-language interaction in a shallow mechanism. In this context, we propose a Deeply coupled Cross-modal Prompt learning (DCP) method based on CLIP. DCP flexibly accommodates the interplay between vision and language with a Cross-Modal Prompt Attention (CMPA) mechanism, which enables the mutual exchange of respective representation through a well-connected multi-head attention module progressively and strongly. We then conduct comprehensive few-shot learning experiments on 11 image classification datasets and analyze the robustness to domain shift as well. Thorough experimental analysis evidently demonstrates the superb few-shot generalization and compelling domain adaption capacity of a well-executed DCP. The code can be found at https://github.com/GingL/CMPA.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
29 more
  1. 2022Visual Prompt Tuningcited by 1,681
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  6. 2019Natural Adversarial Examplescited by 1,408
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021Grounded Language-Image Pre-trainingcited by 1,156
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  14. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  16. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  17. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  18. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  19. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  20. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  21. 2022Prompt Distribution Learningcited by 235
  22. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  23. 2022Unified Vision and Language Prompt Learningcited by 148
  24. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  25. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  26. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  27. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  28. 2022Prompt Tuning with Soft Context Sharing for Vision-Language Modelscited by 13
  29. 2022Declaration-based Prompt Tuning for Visual Question Answeringcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 2; 2026: 1.

11 papers.

  1. 2023Link-Context Learning for Multimodal LLMscited by 24
  2. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  3. 2025Low-rank Prompt Interaction for Continual Vision-Language Retrievalcited by 8
  4. 2024Mixture-of-Prompt-Experts for Multi-modal Semantic Understandingcited by 7
  5. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
  6. 2024m3P: Towards Multimodal Multilingual Translation with Multimodal Promptcited by 6
  7. 2024SynthDoc: Bilingual Documents Synthesis for Visual Document Understandingcited by 4
  8. 2024Advancing Prompt Learning through an External Layercited by 2
  9. 2026HACHIMI: Scalable and Controllable Student Persona Generation via Orchestrated Agentscited by 2
  10. 2023Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Modelscited by 1
  11. 2024What Makes Good Few-shot Examples for Vision-Language Models?cited by 1