mlsanityData to 7 Oct 2026

Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts

arXiv 2410.13030 ↗First posted 16 Oct 2024cs.CV cs.CL cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 30 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

26 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  2. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  3. 2023Consistency-guided Prompt Learning for Vision-Language Models1 Jun
  4. 2023Text encoders bottleneck compositionality in contrastive vision-language models24 May
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct
  6. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  11. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
8 more
  1. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  3. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  4. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  5. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  6. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36
  7. 2022Task Ambiguity in Humans and Language Modelscited by 27
  8. 2024Reasoning about concepts with LLMs: Inconsistencies aboundcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  2. 2025Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Modelscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
  2. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  3. 2025Modeling Variants of Prompts for Vision-Language Models
  4. 2024Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models
  5. 2024Generalizable Prompt Tuning for Vision-Language Models
  6. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  7. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
  8. 2024Progressive Multi-modal Conditional Prompt Tuning