mlsanityData to 14 Oct 2026

Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

arXiv 2211.00768 ↗First posted 1 Nov 2022cs.CL cs.CVcited by 44

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 1 Nov 2022.

Builds on

25 of 51 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  7. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  8. 2020Experience Grounds Languagecited by 242
  9. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  10. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  11. 2019Simple BERT Models for Relation Extraction and Semantic Role Labelingcited by 154
  12. 2022A very preliminary analysis of DALL-E 2cited by 82
13 more
  1. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 73
  2. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 73
  3. 2020UnNatural Language Inferencecited by 69
  4. 2021What Context Features Can Transformer Language Models Use?cited by 50
  5. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
  6. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 34
  7. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  8. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  9. 2022Bugs in the Data: How ImageNet Misrepresents Biodiversitycited by 24
  10. 2020MLMLM: Link Prediction with Mean Likelihood Masked Language Modelcited by 22
  11. 2020Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extractioncited by 18
  12. 2021ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learningcited by 18
  13. 2022To Find Waldo You Need Contextual Cues: Debiasing Who's Waldocited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 14; 2025: 8; 2026: 4.

44 papers.

Sort:

By area:

  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  2. 2023Teaching CLIP to Count to Tencited by 198
  3. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  4. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  5. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  6. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  7. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41
  8. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  9. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  10. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  11. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  12. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
32 more
  1. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  2. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  3. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  4. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  5. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 17
  6. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  7. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  8. 2024DevBench: A multimodal developmental benchmark for language learningcited by 13
  9. 2023A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metricscited by 12
  10. 2023Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understandingcited by 10
  11. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  12. 2024Understanding Depth and Height Perception in Large Visual-Language Modelscited by 8
  13. 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
  14. 2024The Curious Case of Representational Alignment: Unravelling Visio-Linguistic Tasks in Emergent Communicationcited by 7
  15. 2023An Examination of the Compositionality of Large Generative Vision-Language Modelscited by 6
  16. 2024BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrievalcited by 6
  17. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  18. 2025Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Modelscited by 6
  19. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  20. 2024Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspectivecited by 4
  21. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Datacited by 2
  22. 2023Semantic Composition in Visually Grounded Language Modelscited by 1
  23. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterationscited by 1
  24. 2025MASS: Overcoming Language Bias in Image-Text Matchingcited by 1
  25. 2025Adding simple structure at inference improves Vision-Language Compositionalitycited by 1
  26. 2025Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effectcited by 1
  27. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptionscited by 1
  28. 2025LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Modelscited by 1
  29. 2026Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inferencecited by 1
  30. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoningcited by 1
  31. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understandingcited by 0
  32. 2026The Limits of Binding in Dual Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge
  2. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition
  3. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  4. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  5. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs
  6. 2026Eye-Q: A Multilingual Benchmark for Visual Word Puzzle Solving and Image-to-Phrase Reasoning
  7. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?