mlsanityData to 14 Oct 2026

WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models

arXiv 2207.12576 ↗First posted 25 Jul 2022cs.CL cs.AI cs.CVcited by 20

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills. In this work, we introduce WinoGAViL: an online game of vision-and-language associations (e.g., between werewolves and a full moon), used as a dynamic evaluation benchmark. Inspired by the popular card game Codenames, a spymaster gives a textual cue related to several visual candidates, and another player tries to identify them. Human players are rewarded for creating associations that are challenging for a rival AI model but still solvable by other human players. We use the game to collect 3.5K instances, finding that they are intuitive for humans (>90% Jaccard index) but challenging for state-of-the-art AI models, where the best model (ViLT) achieves a score of 52%, succeeding mostly where the cue is visually salient. Our analysis as well as the feedback we collect from players indicate that the collected associations require diverse reasoning skills, including general knowledge, common sense, abstraction, and more. We release the dataset, the code and the interactive game, allowing future data collection that can be used to develop models with better association abilities.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 38 of its 100 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

38 of 100 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022A ConvNet for the 2020scited by 3,579
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
26 more
  1. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  3. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  4. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  5. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  6. 2018Gender Bias in Coreference Resolutioncited by 442
  7. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  8. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  9. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  10. 2019Character Region Awareness for Text Detectioncited by 278
  11. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  12. 2020Adversarial Filters of Dataset Biasescited by 160
  13. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  14. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  15. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  16. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  17. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  18. 2020What is More Likely to Happen Next? Video-and-Language Future Event Predictioncited by 62
  19. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  20. 2021Human-Adversarial Visual Question Answeringcited by 57
  21. 2021Fool Me Twice: Entailment from Wikipedia Gamificationcited by 50
  22. 2020Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioningcited by 36
  23. 2020A Review of Winograd Schema Challenge Datasets and Approachescited by 27
  24. 2017Evaluating Visual Conversational Agents via Cooperative Human-AI Gamescited by 17
  25. 2018Comparing Models of Associative Meaning: An Empirical Investigation of Reference in Simple Language Gamescited by 5
  26. 2015Quizz: Targeted crowdsourcing with a billion (potential) userscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 7; 2024: 9.

20 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  4. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  5. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  6. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  7. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  8. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  9. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  10. 2023Sieve: Multimodal Dataset Pruning Using Image Captioning Modelscited by 33
  11. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24
  12. 2023IRFL: Image Recognition of Figurative Languagecited by 23
8 more
  1. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  2. 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20
  3. 2024HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Textscited by 17
  4. 2024Probabilistic Language-Image Pre-Trainingcited by 16
  5. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  6. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  7. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Modelscited by 6
  8. 2024Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurementcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
  2. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  3. 2026Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
  4. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
  5. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models
  6. 2025VideoGameBench: Can Vision-Language Models complete popular video games?
  7. 2025Are Large Vision Language Models Good Game Players?
  8. 2025WebGames: Challenging General-Purpose Web-Browsing AI Agents