mlsanityData to 7 Oct 2026

CREPE: Can Vision-Language Foundation Models Reason Compositionally?

arXiv 2212.07796 ↗First posted 13 Dec 2022cs.CL cs.CVcited by 177

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, we find that: across 7 architectures trained with 4 algorithms on massive datasets, they struggle at compositionality. To arrive at this conclusion, we introduce a new compositionality evaluation benchmark, CREPE, which measures two important aspects of compositionality identified by cognitive science literature: systematicity and productivity. To measure systematicity, CREPE consists of a test dataset containing over 370K image-text pairs and three different seen-unseen splits. The three splits are designed to test models trained on three popular training datasets: CC-12M, YFCC-15M, and LAION-400M. We also generate 325K, 316K, and 309K hard negative captions for a subset of the pairs. To test productivity, CREPE contains 17K image-text pairs with nine different complexities plus 183K hard negative captions with atomic, swapping and negation foils. The datasets are generated by repurposing the Visual Genome scene graphs and region descriptions and applying handcrafted templates and GPT-3. For systematicity, we find that model performance decreases consistently when novel compositions dominate the retrieval set, with Recall@1 dropping by up to 12%. For productivity, models' retrieval success decays as complexity increases, frequently nearing random chance at high complexity. These results hold regardless of model and training dataset size.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 78 of its 102 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

78 of 102 references.

References · 78

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  10. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  12. 2022Language-driven Semantic Segmentationcited by 711
  13. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  14. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  15. 2019Designing and Interpreting Probes with Control Taskscited by 608
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  17. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  18. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  19. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2015Neural Module Networkscited by 375
  22. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  23. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  24. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  25. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  26. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  27. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  28. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  29. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  30. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  31. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  32. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  33. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  34. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  35. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  36. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  37. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  38. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 30; 2024: 59; 2025: 51; 2026: 36.

177 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  3. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  4. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  5. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  6. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  7. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  8. 2025Vision-Language Models Do Not Understand Negationcited by 69
  9. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  10. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  11. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  12. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
38 more
  1. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  2. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  3. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  4. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 45
  5. 2024Probing Visual Language Priors in VLMscited by 44
  6. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  7. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  8. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
  9. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  10. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  11. 2024N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fieldscited by 34
  12. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1cited by 34
  13. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  14. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  15. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  16. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  17. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  18. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 26
  19. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  20. 2023Prompting Language-Informed Distribution for Compositional Zero-Shot Learningcited by 24
  21. 2025Towards Training-free Anomaly Detection with Vision and Language Foundation Modelscited by 24
  22. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  23. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  24. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  25. 2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20
  26. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  27. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  28. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 18
  29. 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18
  30. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  31. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 17
  32. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  33. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  34. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
  35. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understandingcited by 16
  36. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  37. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  38. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?
  2. 2023The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models
  3. 2024Natural Language Inference Improves Compositionality in Vision-Language Models
  4. 2022A Benchmark for Compositional Visual Reasoning
  5. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
  6. 2025Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generation
  7. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
  8. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias