mlsanityData to 7 Oct 2026

Read-only Prompt Optimization for Vision-Language Few-shot Learning

arXiv 2308.14960 ↗First posted 29 Aug 2023cs.CVcited by 58

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In recent years, prompt tuning has proven effective in adapting pre-trained vision-language models to downstream tasks. These methods aim to adapt the pre-trained models by introducing learnable prompts while keeping pre-trained weights frozen. However, learnable prompts can affect the internal representation within the self-attention module, which may negatively impact performance variance and generalization, especially in data-deficient settings. To address these issues, we propose a novel approach, Read-only Prompt Optimization (RPO). RPO leverages masked attention to prevent the internal representation shift in the pre-trained model. Further, to facilitate the optimization of RPO, the read-only prompts are initialized based on special tokens of the pre-trained model. Our extensive experiments demonstrate that RPO outperforms CLIP and CoCoOp in base-to-new generalization and domain generalization while displaying better robustness. Also, the proposed method achieves better generalization on extremely data-deficient settings, while improving parameter efficiency and computational overhead. Code is available at https://github.com/mlvlab/RPO.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 54 references that are in the map. First posted on arXiv 29 Aug 2023.

Builds on

42 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts4 Dec
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
25 more
  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2021GPT Understands, Toocited by 715
  8. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  9. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  10. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  11. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  12. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  13. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  14. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  15. 2018Bilinear Attention Networkscited by 277
  16. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  17. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  18. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  19. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  20. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  21. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  22. 2022A Broad Study of Pre-training for Domain Generalization and Adaptationcited by 64
  23. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  24. 2022Grounding Visual Representations with Texts for Domain Generalizationcited by 30
  25. 2021The Power of Prompt Tuning for Low-Resource Semantic Parsingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 24; 2025: 22; 2026: 8.

58 papers.

Sort:

By area:

  1. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  2. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  3. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  4. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  5. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  6. 2024Prompt Learning via Meta-Regularizationcited by 26
  7. 2023Large Language Models are Good Prompt Learners for Low-Shot Image Classificationcited by 21
  8. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  9. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  10. 2024Multi-modal Attribute Prompting for Vision-Language Modelscited by 12
  11. 2025Hierarchical Cross-modal Prompt Learning for Vision-Language Modelscited by 12
  12. 2025Logits DeConfusion with CLIP for Few-Shot Learningcited by 11
38 more
  1. 2024Prompt as Free Lunch: Enhancing Diversity in Source-Free Cross-domain Few-shot Learning through Semantic-Guided Promptingcited by 10
  2. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  3. 2024LW2G: Learning Whether to Grow for Prompt-based Continual Learningcited by 9
  4. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  5. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Modelscited by 7
  6. 2024Language-Driven Visual Consensus for Zero-Shot Semantic Segmentationcited by 6
  7. 2024OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentationcited by 6
  8. 2024Conditioned Prompt-Optimization for Continual Deepfake Detectioncited by 6
  9. 2024Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysiscited by 6
  10. 2025Exploring Cross-Modal Flows for Few-Shot Learningcited by 6
  11. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizabilitycited by 5
  12. 2024Quantized Prompt for Efficient Generalization of Vision-Language Modelscited by 5
  13. 2024TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaborationcited by 5
  14. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classificationcited by 5
  15. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  16. 2024Deep Instruction Tuning for Segment Anything Modelcited by 4
  17. 2024Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Modelcited by 4
  18. 2024Adapt PointFormer: 3D Point Cloud Analysis via Adapting 2D Visual Transformerscited by 4
  19. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIPcited by 4
  20. 2025MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Modelscited by 4
  21. 2026Path-Decoupled Hyperbolic Flow Matching for Few-Shot Adaptationcited by 4
  22. 2025Learning Generalizable Prompt for CLIP with Class Similarity Knowledgecited by 3
  23. 2025GLAD: Generalizable Tuning for Vision-Language Modelscited by 3
  24. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  25. 2025BMIP: Bi-directional Modality Interaction Prompt Learning for VLMcited by 2
  26. 2025Modeling Variants of Prompts for Vision-Language Modelscited by 2
  27. 2025Improving Personalized Search with Regularized Low-Rank Parameter Updatescited by 2
  28. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  29. 2023CLAMP: Contrastive LAnguage Model Prompt-tuningcited by 1
  30. 2024Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensemblingcited by 1
  31. 2025Towards Robust Real-World Multivariate Time Series Forecasting: A Unified Framework for Dependency, Asynchrony, and Missingnesscited by 1
  32. 2025VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansioncited by 1
  33. 2025AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learningcited by 1
  34. 2024Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularizationcited by 0
  35. 2025TLAC: Two-stage LMM Augmented CLIP for Zero-Shot Classificationcited by 0
  36. 2025Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrievalcited by 0
  37. 2025AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decompositioncited by 0
  38. 2025Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysiscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
  3. 2024Revisiting Prompt Pretraining of Vision-Language Models
  4. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  5. 2025MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
  6. 2024Generalizable Prompt Tuning for Vision-Language Models
  7. 2024IPO: Interpretable Prompt Optimization for Vision-Language Models
  8. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models