mlsanityData to 14 Oct 2026

Towards Robust Prompts on Vision-Language Models

arXiv 2304.08479 ↗First posted 17 Apr 2023cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the advent of vision-language models (VLMs) that can perform in-context and prompt-based learning, how can we design prompting approaches that robustly generalize to distribution shift and can be used on novel classes outside the support set of the prompts? In this work, we first define two types of robustness to distribution shift on VLMs, namely, robustness on base classes (the classes included in the support set of prompts) and robustness on novel classes. Then, we study the robustness of existing in-context learning and prompt learning approaches, where we find that prompt learning performs robustly on test images from base classes, while it does not generalize well on images from novel classes. We propose robust prompt learning by integrating multiple-scale image features into the prompt, which improves both types of robustness. Comprehensive experiments are conducted to study the defined robustness on six benchmarks and show the effectiveness of our proposal.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 54 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

47 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
31 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  13. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  15. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  16. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  17. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  18. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  19. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  20. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  21. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  22. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  23. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  24. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  25. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  26. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 77
  27. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  28. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 30
  29. 2019Improving the Robustness of Capsule Networks to Image Affine Transformationscited by 21
  30. 2021Capsule Network is Not More Robust than Convolutional Networkcited by 12
  31. 2022Do DALL-E and Flamingo Understand Each Other?cited by 7

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 0; 2025: 2.

7 papers.

  1. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  2. 2023Reliable Evaluation of Adversarial Transferabilitycited by 13
  3. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  4. 2023Universal Domain Adaptation from Foundation Models: A Baseline Studycited by 10
  5. 2025Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Controlcited by 5
  6. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Designcited by 3
  7. 2023Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attackscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Modeling Variants of Prompts for Vision-Language Models
  2. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  3. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  4. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
  5. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  6. 2024Revisiting Prompt Pretraining of Vision-Language Models
  7. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  8. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration