# MMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Contextual Reasoning

arXiv: https://arxiv.org/abs/2603.01055 · HTML: https://mlsanity.com/paper/2603.01055/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 1 Mar 2026.

## Builds on (most cited first, 22 of 22)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [TransOMCS: From Linguistic Graphs to Commonsense Knowledge](https://mlsanity.com/paper/2005.00206/) (2020)
- [A Survey on Open Information Extraction](https://mlsanity.com/paper/1806.05599/) (2018)
- [CSKG: The CommonSense Knowledge Graph](https://mlsanity.com/paper/2012.11490/) (2020)
- [VisualSem: A High-quality Knowledge Graph for Vision and Language](https://mlsanity.com/paper/2008.09150/) (2020)
- ["I'm Not Mad": Commonsense Implications of Negation and Contradiction](https://mlsanity.com/paper/2104.06511/) (2021)
- [Refined Commonsense Knowledge from Large-Scale Web Contents](https://mlsanity.com/paper/2112.04596/) (2021)
- [Improving Image Captioning by Leveraging Knowledge Graphs](https://mlsanity.com/paper/1901.08942/) (2019)
- [RoViST:Learning Robust Metrics for Visual Storytelling](https://mlsanity.com/paper/2205.03774/) (2022)
- [Completeness, Recall, and Negation in Open-World Knowledge Bases: A Survey](https://mlsanity.com/paper/2305.05403/) (2023)
- [SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling](https://mlsanity.com/paper/2402.00319/) (2024)

## Abstract (arXiv)

We present MMCOMET, the first multimodal commonsense knowledge graph (MMKG) that integrates physical, social, and eventive knowledge. MMCOMET extends the ATOMIC2020 knowledge graph to include a visual dimension, through an efficient image retrieval process, resulting in over 900K multimodal triples. This new resource addresses a major limitation of existing MMKGs in supporting complex reasoning tasks like image captioning and storytelling. Through a standard visual storytelling experiment, we show that our holistic approach enables the generation of richer, coherent, and contextually grounded stories than those produced using text-only knowledge. This resource establishes a new foundation for multimodal commonsense reasoning and narrative generation.