mlsanityData to 7 Oct 2026

Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

arXiv 2208.08340 ↗First posted 17 Aug 2022cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the emergence of large pre-trained vison-language model like CLIP, transferable representations can be adapted to a wide range of downstream tasks via prompt tuning. Prompt tuning tries to probe the beneficial information for downstream tasks from the general knowledge stored in the pre-trained model. A recently proposed method named Context Optimization (CoOp) introduces a set of learnable vectors as text prompt from the language side. However, tuning the text prompt alone can only adjust the synthesized "classifier", while the computed visual features of the image encoder can not be affected , thus leading to sub-optimal solutions. In this paper, we propose a novel Dual-modality Prompt Tuning (DPT) paradigm through learning text and visual prompts simultaneously. To make the final image feature concentrate more on the target visual concept, a Class-Aware Visual Prompt Tuning (CAVPT) scheme is further proposed in our DPT, where the class-aware visual prompt is generated dynamically by performing the cross attention between text prompts features and image patch token embeddings to encode both the downstream task-related information and visual instance information. Extensive experimental results on 11 datasets demonstrate the effectiveness and generalization ability of the proposed method. Our code is available in https://github.com/fanrena/DPT.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 67 references that are in the map. First posted on arXiv 17 Aug 2022.

Builds on

44 of 67 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  2. 2022Prompt Distribution Learning6 May
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  12. 2013Describing Textures in the Wildcited by 2,110
25 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  8. 2019How Can We Know What Language Models Know?cited by 953
  9. 2021Learning to Prompt for Continual Learningcited by 938
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  12. 2017Learning multiple visual domains with residual adapterscited by 553
  13. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  14. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  15. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  18. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  19. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  20. 2022Neural Prompt Searchcited by 133
  21. 2022Can Language Understand Depth?cited by 66
  22. 2022Prompting for Multi-Modal Trackingcited by 58
  23. 2019Low-Rank Pairwise Alignment Bilinear Network For Few-Shot Fine-Grained Image Classificationcited by 36
  24. 2022Prompt Tuning for Generative Multimodal Pretrained Modelscited by 25
  25. 2022Multi-level Second-order Few-shot Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 18; 2025: 18; 2026: 4.

51 papers.

Sort:

By area:

  1. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  2. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
  3. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  4. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  5. 2022Learning Domain Invariant Prompt for Vision-Language Modelscited by 35
  6. 2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
  7. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  8. 2025A Survey on Efficient Vision-Language Modelscited by 19
  9. 2024Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approachcited by 17
  10. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  11. 2024Frequency-Guided Spatial Adaptation for Camouflaged Object Detectioncited by 15
  12. 2022Prompt Tuning with Soft Context Sharing for Vision-Language Modelscited by 13
38 more
  1. 2023When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence Systemcited by 13
  2. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  3. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
  4. 2024Subgraph-level Universal Prompt Tuningcited by 9
  5. 2024Vision-Language Consistency Guided Multi-modal Prompt Learning for Blind AI Generated Image Quality Assessmentcited by 9
  6. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  7. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  8. 2025Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptationcited by 7
  9. 2023Retrieval-Enhanced Visual Prompt Learning for Few-shot Classificationcited by 6
  10. 2024FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentationcited by 6
  11. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  12. 2024MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusioncited by 5
  13. 2025Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detectorcited by 5
  14. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Modelscited by 4
  15. 2023UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Promptscited by 3
  16. 2024HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adaptercited by 3
  17. 2024RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-trainingcited by 3
  18. 2025SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptationcited by 3
  19. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  20. 2023M-Tuning: Prompt Tuning with Mitigated Label Bias in Open-Set Scenarioscited by 2
  21. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  22. 2025Improving Personalized Search with Regularized Low-Rank Parameter Updatescited by 2
  23. 2025Simulate, Refocus and Ensemble: An Attention-Refocusing Scheme for Domain Generalizationcited by 2
  24. 2023Adaptive Multi-Modality Prompt Learningcited by 1
  25. 2024Enhancing Vision-Language Models Generalization via Diversity-Driven Novel Feature Synthesiscited by 1
  26. 2024Lightweight Model Pre-training via Language Guided Knowledge Distillationcited by 1
  27. 2024SDPT: Synchronous Dual Prompt Tuning for Fusion-based Visual-Language Pre-trained Modelscited by 1
  28. 2024MoPD: Mixture-of-Prompts Distillation for Vision-Language Modelscited by 1
  29. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
  30. 2024RESTORE: Towards Feature Shift for Vision-Language Prompt Learningcited by 0
  31. 2024Dual-Modal Prompting for Sketch-Based Image Retrievalcited by 0
  32. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Modelscited by 0
  33. 2025ChordPrompt: Orchestrating Cross-Modal Prompt Synergy for Multi-Domain Incremental Learning in CLIPcited by 0
  34. 2025MPT: Motion Prompt Tuning for Micro-Expression Recognitioncited by 0
  35. 2025RASALoRE: Region Aware Spatial Attention with Location-based Random Embeddings for Weakly Supervised Anomaly Detection in Brain MRI Scanscited by 0
  36. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  37. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weightingcited by 0
  38. 2026BadBone: Backdoor Attacks Against Backbone Models in Visual Prompt Learningcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SEP: Self-Enhanced Prompt Tuning for Visual-Language Model
  2. 2024Progressive Multi-modal Conditional Prompt Tuning
  3. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  4. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  5. 2024Generalizable Prompt Tuning for Vision-Language Models
  6. 2023Task-Oriented Multi-Modal Mutual Leaning for Vision-Language Models
  7. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  8. 2023Distribution-Aware Prompt Tuning for Vision-Language Models