# VCD: A Dataset for Visual Commonsense Discovery in Images

arXiv: https://arxiv.org/abs/2402.17213 · HTML: https://mlsanity.com/paper/2402.17213/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [ASER: A Large-scale Eventuality Knowledge Graph](https://mlsanity.com/paper/1905.00270/) (2019)
- [Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation](https://mlsanity.com/paper/1707.09423/) (2017)
- [TransOMCS: From Linguistic Graphs to Commonsense Knowledge](https://mlsanity.com/paper/2005.00206/) (2020)
- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [Things not Written in Text: Exploring Spatial Commonsense from Visual Signals](https://mlsanity.com/paper/2203.08075/) (2022)
- [Visual Commonsense in Pretrained Unimodal and Multimodal Models](https://mlsanity.com/paper/2205.01850/) (2022)
- [VisualSem: A High-quality Knowledge Graph for Vision and Language](https://mlsanity.com/paper/2008.09150/) (2020)
- [DISCOS: Bridging the Gap between Discourse Knowledge and Commonsense Knowledge](https://mlsanity.com/paper/2101.00154/) (2021)
- [Can Language Models Understand Physical Concepts?](https://mlsanity.com/paper/2305.14057/) (2023)
- [Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templates](https://mlsanity.com/paper/1711.06821/) (2017)
- [Intrinsic Physical Concepts Discovery with Object-Centric Predictive Models](https://mlsanity.com/paper/2303.01869/) (2023)
- [Dense-ATOMIC: Towards Densely-connected ATOMIC with High Knowledge Coverage and Massive Multi-hop Paths](https://mlsanity.com/paper/2210.07621/) (2022)
- [VIPHY: Probing "Visible" Physical Commonsense Knowledge](https://mlsanity.com/paper/2209.07000/) (2022)
- [ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories](https://mlsanity.com/paper/2305.15028/) (2023)

## Built on by (most cited first, 5 of 5)

- [MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Models](https://mlsanity.com/paper/2407.17379/) (2024)
- [Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs](https://mlsanity.com/paper/2403.07398/) (2024)
- [CodeGraph: Enhancing Graph Reasoning of LLMs with Code](https://mlsanity.com/paper/2408.13863/) (2024)
- [KNOWCOMP POKEMON Team at DialAM-2024: A Two-Stage Pipeline for Detecting Relations in Dialogical Argument Mining](https://mlsanity.com/paper/2407.19740/) (2024)
- [AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?](https://mlsanity.com/paper/2509.17641/) (2025)

## Abstract (arXiv)

Visual commonsense plays a vital role in understanding and reasoning about the visual world. While commonsense knowledge bases like ConceptNet provide structured collections of general facts, they lack visually grounded representations. Scene graph datasets like Visual Genome, though rich in object-level descriptions, primarily focus on directly observable information and lack systematic categorization of commonsense knowledge. We present Visual Commonsense Dataset (VCD), a large-scale dataset containing over 100,000 images and 14 million object-commonsense pairs that bridges this gap. VCD introduces a novel three-level taxonomy for visual commonsense, integrating both Seen (directly observable) and Unseen (inferrable) commonsense across Property, Action, and Space aspects. Each commonsense is represented as a triple where the head entity is grounded to object bounding boxes in images, enabling scene-dependent and object-specific visual commonsense representation. To demonstrate VCD's utility, we develop VCM, a generative model that combines a vision-language model with instruction tuning to discover diverse visual commonsense from images. Extensive evaluations demonstrate both the high quality of VCD and its value as a resource for advancing visually grounded commonsense understanding and reasoning. Our dataset and code will be released on https://github.com/NUSTM/VCD.