mlsanityData to 14 Oct 2026

SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations

arXiv 2406.11171 ↗First posted 17 Jun 2024cs.CV cs.CL cs.LGcited by 42

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite their remarkable successes, state-of-the-art large language models (LLMs), including vision-and-language models (VLMs) and unimodal language models (ULMs), fail to understand precise semantics. For example, semantically equivalent sentences expressed using different lexical compositions elicit diverging representations. The degree of this divergence and its impact on encoded semantics is not very well understood. In this paper, we introduce the SUGARCREPE++ dataset to analyze the sensitivity of VLMs and ULMs to lexical and semantic alterations. Each sample in SUGARCREPE++ dataset consists of an image and a corresponding triplet of captions: a pair of semantically equivalent but lexically different positive captions and one hard negative caption. This poses a 3-way semantic (in)equivalence problem to the language models. We comprehensively evaluate VLMs and ULMs that differ in architecture, pre-training objectives and datasets to benchmark the performance of SUGARCREPE++ dataset. Experimental results highlight the difficulties of VLMs in distinguishing between lexical and semantic variations, particularly in object attributes and spatial relations. Although VLMs with larger pre-training datasets, model sizes, and multiple pre-training objectives achieve better performance on SUGARCREPE++, there is a significant opportunity for improvement. We show that all the models which achieve better performance on compositionality datasets need not perform equally well on SUGARCREPE++, signifying that compositionality alone may not be sufficient for understanding semantic and lexical alterations. Given the importance of the property that the SUGARCREPE++ dataset targets, it serves as a new challenge to the vision-and-language community.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 119 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

72 of 119 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense30 Oct
  2. 2023Revisiting the Role of Language Priors in Vision-Language Models2 Jun
  3. 2023RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models21 Apr
  4. 2023Probing Conceptual Understanding of Large Visual-Language Models7 Apr
  5. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov

Other references · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  13. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  14. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  15. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  16. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  17. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  18. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  19. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  20. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  21. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  22. 2021Editing Factual Knowledge in Language Modelscited by 553
  23. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  24. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  25. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  26. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 429
  27. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  28. 2023Improving Text Embeddings with Large Language Modelscited by 303
  29. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  30. 2022Visual Spatial Reasoningcited by 293
  31. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  32. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 223
  33. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  34. 2021Decoupling Zero-Shot Semantic Segmentationcited by 199
  35. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  36. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  37. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  38. 2023AnglE-optimized Text Embeddingscited by 99

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 19; 2026: 17.

42 papers.

Sort:

By area:

  1. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  2. 2025Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinkingcited by 16
  3. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  4. 2025CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representationcited by 14
  5. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  6. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 10
  7. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  8. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  9. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  10. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  11. 2025Evaluating Compositional Generalisation in VLMs and Diffusion Modelscited by 5
  12. 2024Resolving Lexical Bias in Model Editingcited by 4
30 more
  1. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
  2. 2024Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Modelscited by 3
  3. 2024Sensitivity of Generative VLMs to Semantically and Lexically Altered Promptscited by 2
  4. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  5. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  6. 2026Reward Prediction with Factorized World Statescited by 2
  7. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  8. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  9. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  10. 2026PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensemblescited by 1
  11. 2026On the Persistent Effects of Lexicality in Large Language Modelscited by 1
  12. 2024On Debiasing Text Embeddings Through Context Injectioncited by 0
  13. 2024A New Method to Capturing Compositional Knowledge in Linguistic Spacecited by 0
  14. 2025Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Modelscited by 0
  15. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMscited by 0
  16. 2025What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenescited by 0
  17. 2025Contrastive vision-language learning with paraphrasing and negationcited by 0
  18. 2026Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Modelscited by 0
  19. 2026MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignmentcited by 0
  20. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0
  21. 2026Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Familycited by 0
  22. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0
  23. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  24. 2026Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weightingcited by 0
  25. 2026Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learningcited by 0
  26. 2026The Limits of Binding in Dual Encoderscited by 0
  27. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representationscited by 0
  28. 2026CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillationcited by 0
  29. 2026BindCLIP: One Balanced Coupling For Compositional Vision Language Scoringcited by 0
  30. 2026Can Motion-Language Models Ground Structure? STRIDE for Evaluating the Evaluatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
  2. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  3. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  4. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  5. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  6. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  7. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  8. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models