mlsanityData to 14 Oct 2026

Modeling Variants of Prompts for Vision-Language Models

arXiv 2503.08229 ↗First posted 11 Mar 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large pre-trained vision-language models (VLMs) offer a promising approach to leveraging human language for enhancing downstream tasks. However, VLMs such as CLIP face significant limitation: its performance is highly sensitive to prompt template design. Although prompt learning methods can address the sensitivity issue by replacing natural language prompts with learnable ones, they are incomprehensible to humans. Ensuring consistent performance across various prompt templates enables models to adapt seamlessly to diverse phrasings, enhancing their ability to handle downstream tasks without requiring extensive prompt engineering. In this work, we introduce the RobustPrompt Benchmark, a systematic benchmark to evaluate robustness to different prompt templates for VLMs. It includes a dataset with hundreds of carefully designed prompt templates, divided into six types, covering a wide variety of commonly used templates. Beside the benchmark, we propose Modeling Variants of Prompts (MVP), a simple yet effective method that mitigates sensitivity by modeling variants of prompt structures. The innovation of MVP lies in decoupling prompts into templates and class names, and using Variational Autoencoders (VAE) to model the distribution of diverse prompt structures. Experiments across 11 datasets demonstrate that MVP can greatly enhance model robustness to variations in input prompts without a drop in performance. The code is available at https://github.com/liaolea/MVP.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 37 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

23 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learning29 Aug
  2. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts14 Jul
  3. 2022Prompt Distribution Learning6 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
4 more
  1. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  2. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  3. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  4. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ORION: ORthonormal Text Encoding for Universal VLM AdaptatIONcited by 1
  2. 2026OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Robust Prompts on Vision-Language Models
  2. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
  3. 2025Integrated Structural Prompt Learning for Vision-Language Models
  4. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  5. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  6. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  7. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  8. 2024Generalizable Prompt Tuning for Vision-Language Models