mlsanityData to 14 Oct 2026

Visual Commonsense in Pretrained Unimodal and Multimodal Models

arXiv 2205.01850 ↗First posted 4 May 2022cs.CL cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple. Text and image corpora, being subject to reporting bias, represent this world-knowledge to varying degrees of faithfulness. In this paper, we investigate to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. To that end, we create the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects. We validate this dataset by showing that our grounded color data correlates much better than ungrounded text-only data with crowdsourced color judgments provided by Paik et al. (2021). We then use our dataset to evaluate pretrained unimodal models and multimodal models. Our results indicate that multimodal models better reconstruct attribute distributions, but are still subject to reporting bias. Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 4 May 2022.

Builds on

27 of 40 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
15 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  3. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  4. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  5. 2020Experience Grounds Languagecited by 242
  6. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  7. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  8. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 94
  9. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  10. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 72
  11. 2015Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labelscited by 64
  12. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  13. 2020Overcoming Language Priors with Self-supervised Learning for Visual Question Answeringcited by 43
  14. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  15. 2020Probing Neural Language Models for Human Tacit Assumptionscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 11; 2023: 11; 2024: 10; 2025: 7.

39 papers.

Sort:

By area:

  1. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  4. 2024Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?cited by 52
  5. 2023Large language models predict human sensory judgments across six modalitiescited by 48
  6. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
  7. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  8. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  9. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  10. 2024LLMGeo: Benchmarking Large Language Models on Image Geolocation In-the-wildcited by 21
  11. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 20
  12. 2023Can Language Models Understand Physical Concepts?cited by 19
27 more
  1. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  2. 2022Visually-Augmented Language Modelingcited by 18
  3. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
  4. 2023On the Difference of BERT-style and CLIP-style Text Encoderscited by 15
  5. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  6. 2023Kiki or Bouba? Sound Symbolism in Vision-and-Language Modelscited by 14
  7. 2024VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Modelscited by 8
  8. 2022Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colourcited by 5
  9. 2022Visually-augmented pretrained language models for NLP tasks without imagescited by 5
  10. 2023Substance or Style: What Does Your Image Embedding Know?cited by 5
  11. 2024VCD: A Dataset for Visual Commonsense Discovery in Imagescited by 5
  12. 2025The Zero Body Problem: Probing LLM Use of Sensory Languagecited by 5
  13. 2022VIPHY: Probing "Visible" Physical Commonsense Knowledgecited by 4
  14. 2023Understanding Natural Language Understanding Systems. A Critical Analysiscited by 4
  15. 2023ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categoriescited by 4
  16. 2025VAQUUM: Are Vague Quantifiers Grounded in Visual Data?cited by 4
  17. 2025Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Modelscited by 3
  18. 2022Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Modelscited by 2
  19. 2024AudioBERT: Audio Knowledge Augmented Language Modelcited by 2
  20. 2024System 2 Reasoning Capabilities Are Nighcited by 2
  21. 2025Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Modelscited by 2
  22. 2022Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Predictioncited by 1
  23. 2024WinoViz: Probing Visual Properties of Objects Under Different Statescited by 1
  24. 2025Estimating Commonsense Plausibility through Semantic Shiftscited by 1
  25. 2025AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?cited by 1
  26. 2024Improving the Efficiency of Visually Augmented Language Modelscited by 0
  27. 2025Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?cited by 0