mlsanityData to 7 Oct 2026

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

arXiv 2303.07274 ↗First posted 13 Mar 2023cs.CV cs.AI cs.CLcited by 79

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field. Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. Data, models and code are available at the project website: whoops-benchmark.github.io

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

38 of 63 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
26 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  12. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  13. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  14. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  15. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  16. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  17. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  18. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  19. 2022All You May Need for VQA are Image Captionscited by 65
  20. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  21. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  22. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  23. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  24. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  25. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  26. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 32; 2025: 11; 2026: 15.

79 papers.

Sort:

By area:

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  5. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  6. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  7. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  8. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  9. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  10. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  11. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  12. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
38 more
  1. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  2. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  3. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  4. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
  5. 2024Probing Visual Language Priors in VLMscited by 44
  6. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  7. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  8. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Modelscited by 29
  9. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  10. 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sensecited by 27
  11. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  12. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  13. 2023IRFL: Image Recognition of Figurative Languagecited by 23
  14. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  15. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  16. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  17. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  18. 2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
  19. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  20. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 14
  21. 2025Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captionscited by 13
  22. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarkscited by 12
  23. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  24. 2024ColorSwap: A Color and Word Order Dataset for Multimodal Evaluationcited by 11
  25. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictionscited by 11
  26. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  27. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  28. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  29. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  30. 2024Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusionscited by 10
  31. 2025ExDDV: A New Dataset for Explainable Deepfake Detection in Videocited by 10
  32. 2023TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Modelscited by 9
  33. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
  34. 2023Fully Authentic Visual Question Answering Dataset from Online Communitiescited by 9
  35. 2024YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Modelscited by 9
  36. 2025Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumblecited by 9
  37. 2025Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learningcited by 9
  38. 2025Image Generation from Contextually-Contradictory Promptscited by 9

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding
  2. 2026Enhancing Zero-shot Commonsense Reasoning by Integrating Visual Knowledge via Machine Imagination
  3. 2024The Visual Counter Turing Test (VCT2): A Benchmark for Evaluating AI-Generated Image Detection and the Visual AI Index (VAI)
  4. 2026Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Odd
  5. 2025COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark
  6. 2025Spot The Ball: A Benchmark for Visual Social Inference
  7. 2022CoSIm: Commonsense Reasoning for Counterfactual Scene Imagination
  8. 2024Zero-shot Commonsense Reasoning over Machine Imagination