# KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning

arXiv: https://arxiv.org/abs/2012.07000 · HTML: https://mlsanity.com/paper/2012.07000/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 18 of 18)

- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [Localized Symbolic Knowledge Distillation for Visual Commonsense Models](https://mlsanity.com/paper/2312.04837/) (2023)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection](https://mlsanity.com/paper/2112.06888/) (2021)
- [WhyAct: Identifying Action Reasons in Lifestyle Vlogs](https://mlsanity.com/paper/2109.02747/) (2021)
- [Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR](https://mlsanity.com/paper/2405.16934/) (2024)
- [Learning to Agree on Vision Attention for Visual Commonsense Reasoning](https://mlsanity.com/paper/2302.02117/) (2023)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoning](https://mlsanity.com/paper/2404.13847/) (2024)
- [Unified Video-Language Pre-training with Synchronized Audio](https://mlsanity.com/paper/2405.07202/) (2024)
- [Local and Global Context-Based Pairwise Models for Sentence Ordering](https://mlsanity.com/paper/2110.04291/) (2021)
- [Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions](https://mlsanity.com/paper/2405.11145/) (2024)
- [CAVL: Learning Contrastive and Adaptive Representations of Vision and Language](https://mlsanity.com/paper/2304.04399/) (2023)
- [Improving vision-language alignment with graph spiking hybrid Networks](https://mlsanity.com/paper/2501.19069/) (2025)

## Abstract (arXiv)

Reasoning is a critical ability towards complete visual understanding. To develop machine with cognition-level visual understanding and reasoning abilities, the visual commonsense reasoning (VCR) task has been introduced. In VCR, given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. The methods adopting the powerful BERT model as the backbone for learning joint representation of image content and natural language have shown promising improvements on VCR. However, none of the existing methods have utilized commonsense knowledge in visual commonsense reasoning, which we believe will be greatly helpful in this task. With the support of commonsense knowledge, complex questions even if the required information is not depicted in the image can be answered with cognitive reasoning. Therefore, we incorporate commonsense knowledge into the cross-modal BERT, and propose a novel Knowledge Enhanced Visual-and-Linguistic BERT (KVL-BERT for short) model. Besides taking visual and linguistic contents as input, external commonsense knowledge extracted from ConceptNet is integrated into the multi-layer Transformer. In order to reserve the structural information and semantic representation of the original sentence, we propose using relative position embedding and mask-self-attention to weaken the effect between the injected commonsense knowledge and other unrelated components in the input sequence. Compared to other task-specific models and general task-agnostic pre-training models, our KVL-BERT outperforms them by a large margin.