mlsanityData to 14 Oct 2026

SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality

arXiv 2306.14610 ↗First posted 26 Jun 2023cs.CV cs.CL cs.LGcited by 243

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In the last year alone, a surge of new benchmarks to measure compositional understanding of vision-language models have permeated the machine learning ecosystem. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. Surprisingly, we find significant biases in all these benchmarks rendering them hackable. This hackability is so dire that blind models with no access to the image outperform state-of-the-art vision-language models. To remedy this rampant vulnerability, we introduce SugarCrepe, a new benchmark for vision-language compositionality evaluation. We employ large language models, instead of rule-based templates used in previous benchmarks, to generate fluent and sensical hard negatives, and utilize an adversarial refinement mechanism to maximally reduce biases. We re-evaluate state-of-the-art models and recently proposed compositionality inducing strategies, and find that their improvements were hugely overestimated, suggesting that more innovation is needed in this important direction. We release SugarCrepe and the code for evaluation at: https://github.com/RAIVNLab/sugar-crepe.

Key facts. Cited by 243 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 54 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

38 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Image Captioners Are Scalable Vision Learners Too13 Jun
  2. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality23 May
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  4. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
22 more
  1. 2023A Watermark for Large Language Modelscited by 725
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  4. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  5. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  8. 2016Visual Relationship Detection with Language Priorscited by 334
  9. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  10. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  11. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  12. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  13. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  14. 2020Adversarial Filters of Dataset Biasescited by 160
  15. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  16. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  17. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  18. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  19. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  20. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  21. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  22. 2023Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biasescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 73; 2025: 72; 2026: 88.

243 papers.

Sort:

By area:

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  3. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  4. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  5. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 103
  6. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  7. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  8. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  9. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  10. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  11. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  12. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
38 more
  1. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 46
  2. 2024Probing Visual Language Priors in VLMscited by 45
  3. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  4. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  5. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  6. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  7. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scalingcited by 31
  8. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 31
  9. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 31
  10. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 31
  11. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  12. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  13. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 28
  14. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  15. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  16. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  17. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 22
  18. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  19. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  20. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21
  21. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  22. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 18
  23. 2024HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoningcited by 18
  24. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  25. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  26. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  27. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 17
  28. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  29. 2024Pre-trained Vision-Language Models Learn Discoverable Visual Conceptscited by 16
  30. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understandingcited by 16
  31. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Modelscited by 15
  32. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Modelscited by 15
  33. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  34. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  35. 2025MIEB: Massive Image Embedding Benchmarkcited by 15
  36. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  37. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  38. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14

50 most cited of 243 All 243 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  3. 2023COLA: A Benchmark for Compositional Text-to-image Retrieval
  4. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
  5. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
  6. 2023Scalable Performance Analysis for Vision-Language Models
  7. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  8. 2025Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models