mlsanityData to 7 Oct 2026

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

arXiv 2204.03162 ↗First posted 7 Apr 2022cs.CV cs.CLcited by 523

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground. Given two images and two captions, the goal is to match them correctly - but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set of fine-grained tags to assist in analyzing model performance. We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance. Evidently, these models are not as skilled at visio-linguistic compositional reasoning as we might have hoped. We perform an extensive analysis to obtain insights into how future work might try to mitigate these models' shortcomings. We aim for Winoground to serve as a useful evaluation set for advancing the state of the art and driving further progress in the field. The dataset is available at https://huggingface.co/datasets/facebook/winoground.

Key facts. Cited by 523 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 69 of its 92 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

69 of 92 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
38 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  15. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  16. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  17. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  18. 2018Gender Bias in Coreference Resolutioncited by 438
  19. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  20. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  21. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  22. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  23. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  24. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  25. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  26. 2019Connecting Vision and Language with Localized Narrativescited by 225
  27. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  28. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  29. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  30. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  31. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  32. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  33. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  34. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  35. 2019Exploring Hate Speech Detection in Multimodal Publicationscited by 103
  36. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  37. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  38. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 35; 2023: 106; 2024: 146; 2025: 109; 2026: 127.

523 papers.

Sort:

By area:

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  4. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  5. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  6. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  7. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  8. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  9. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  10. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  11. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  12. 2023Teaching CLIP to Count to Tencited by 194
38 more
  1. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  2. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  3. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  4. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  5. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  6. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  7. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  8. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  9. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  10. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  11. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  12. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  13. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  14. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  15. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  16. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  17. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  18. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  19. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  20. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  21. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  22. 2022A very preliminary analysis of DALL-E 2cited by 82
  23. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  24. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  25. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  26. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  27. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  28. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  29. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  30. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  31. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  32. 2025Towards Understanding Camera Motions in Any Videocited by 62
  33. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  34. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  35. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  36. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  37. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  38. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55

50 most cited of 523 All 523 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  2. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  3. 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models
  4. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
  5. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
  6. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples
  7. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models
  8. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning