mlsanityData to 7 Oct 2026

Unified Vision and Language Prompt Learning

arXiv 2210.07225 ↗First posted 13 Oct 2022cs.CV cs.AIcited by 145

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt tuning, a parameter- and data-efficient transfer learning paradigm that tunes only a small number of parameters in a model's input space, has become a trend in the vision community since the emergence of large vision-language models like CLIP. We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that none of the unimodal prompt tuning methods performs consistently well: text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities. Extensive experiments on over 11 vision datasets show that UPT achieves a better trade-off than the unimodal counterparts on few-shot learning benchmarks, as well as on domain generalization benchmarks. Code and models will be released to facilitate future research.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 43 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

31 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Prompt Distribution Learning6 May
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models24 Sep
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
15 more
  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  8. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  10. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  11. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  12. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  13. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  14. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  15. 2022Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 54; 2024: 48; 2025: 31; 2026: 9.

145 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  3. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  4. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  5. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 100
  6. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  7. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  8. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  9. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Modelcited by 75
  10. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  11. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  12. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 59
38 more
  1. 2022Multitask Vision-Language Prompt Tuningcited by 51
  2. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognitioncited by 51
  3. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  4. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  5. 2023MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphscited by 40
  6. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  7. 2023Distribution-Aware Prompt Tuning for Vision-Language Modelscited by 35
  8. 2023Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizercited by 35
  9. 2023Visual Tuningcited by 34
  10. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  11. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 33
  12. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  13. 2024Non-Homophilic Graph Pre-Training and Prompt Learningcited by 31
  14. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuningcited by 30
  15. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  16. 2024Visual Fourier Prompt Tuningcited by 30
  17. 2025Visual Instance-aware Prompt Tuningcited by 29
  18. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  19. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  20. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  21. 2023Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learningcited by 26
  22. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
  23. 2024DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrievalcited by 26
  24. 2024SeD: Semantic-Aware Discriminator for Image Super-Resolutioncited by 26
  25. 2024Prompt Learning via Meta-Regularizationcited by 26
  26. 2023SelfPromer: Self-Prompt Dehazing Transformers with Depth-Consistencycited by 24
  27. 2023Link-Context Learning for Multimodal LLMscited by 24
  28. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
  29. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  30. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  31. 2023DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotationscited by 22
  32. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
  33. 2023SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channelscited by 20
  34. 2024Cascade Prompt Learning for Vision-Language Model Adaptationcited by 19
  35. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
  36. 2023Federated Prompt Learning for Weather Foundation Models on Devicescited by 17
  37. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  38. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Modelscited by 17

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Domain Invariant Prompt for Vision-Language Models
  2. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
  3. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
  5. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Models
  6. 2023Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment
  7. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  8. 2024LPT: Less-overfitting Prompt Tuning for Vision-Language Model