mlsanityData to 7 Oct 2026

VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge

arXiv 2210.13626 ↗First posted 24 Oct 2022cs.CV cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

There has been a growing interest in solving Visual Question Answering (VQA) tasks that require the model to reason beyond the content present in the image. In this work, we focus on questions that require commonsense reasoning. In contrast to previous methods which inject knowledge from static knowledge bases, we investigate the incorporation of contextualized knowledge using Commonsense Transformer (COMET), an existing knowledge model trained on human-curated knowledge bases. We propose a method to generate, select, and encode external commonsense knowledge alongside visual and textual cues in a new pre-trained Vision-Language-Commonsense transformer model, VLC-BERT. Through our evaluation on the knowledge-intensive OK-VQA and A-OKVQA datasets, we show that VLC-BERT is capable of outperforming existing models that utilize static knowledge bases. Furthermore, through a detailed analysis, we explain which questions benefit, and which don't, from contextualized commonsense knowledge from COMET.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

42 of 50 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2015Fast R-CNNcited by 2,930
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
30 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  8. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  9. 2016Visual Dialogcited by 377
  10. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  11. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  13. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  14. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  15. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  16. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  17. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  18. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  19. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  20. 2021WebQA: Multihop and Multimodal QAcited by 109
  21. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  22. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  23. 2018Improving Question Answering by Commonsense-Based Pre-Trainingcited by 66
  24. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  25. 2020Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansionscited by 49
  26. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  27. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  28. 2018Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examplescited by 34
  29. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  30. 2021HypoGen: Hyperbole Generation with Commonsense and Counterfactual Knowledgecited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 6; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  2. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  3. 2024Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answeringcited by 15
  4. 2023Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questionscited by 9
  5. 2024Rationale-guided Prompting for Knowledge-based Visual Question Answeringcited by 9
  6. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  7. 2024Neural Multimodal Topic Modeling: A Comprehensive Evaluationcited by 5
  8. 2024Learning from the Web: Language Drives Weakly-Supervised Incremental Learning for Semantic Segmentationcited by 5
  9. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4
  10. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  11. 2024Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answeringcited by 3
  12. 2023COMET-M: Reasoning about Multiple Events in Complex Sentencescited by 2
8 more
  1. 2023Implicit and Explicit Commonsense for Multi-sentence Video Captioningcited by 1
  2. 2025MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answeringcited by 1
  3. 2025LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matchingcited by 1
  4. 2025StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answeringcited by 1
  5. 2025Multi-Modal Feature Fusion for Spatial Morphology Analysis of Traditional Villages via Hierarchical Graph Neural Networkscited by 1
  6. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answeringcited by 1
  7. 2024Enhanced Textual Feature Extraction for Visual Question Answering: A Simple Convolutional Approachcited by 0
  8. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR
  2. 2025NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks
  3. 2024Multimodal Commonsense Knowledge Distillation for Visual Question Answering
  4. 2022Joint Answering and Explanation for Visual Commonsense Reasoning
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  6. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
  7. 2024Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA
  8. 2023ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models