mlsanityData to 7 Oct 2026

Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base

arXiv 2207.13242 ↗First posted 27 Jul 2022cs.CV cs.AI cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Knowledge-based visual question answering (KVQA) task aims to answer questions that require additional external knowledge as well as an understanding of images and questions. Recent studies on KVQA inject an external knowledge in a multi-modal form, and as more knowledge is used, irrelevant information may be added and can confuse the question answering. In order to properly use the knowledge, this study proposes the following: 1) we introduce a novel semantic inconsistency measure computed from caption uncertainty and semantic similarity; 2) we suggest a new external knowledge assimilation method based on the semantic inconsistency measure and apply it to integrate explicit knowledge and implicit knowledge for KVQA; 3) the proposed method is evaluated with the OK-VQA dataset and achieves the state-of-the-art performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 27 Jul 2022.

Builds on

27 of 44 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
15 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2018Bilinear Attention Networkscited by 277
  7. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  8. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  9. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  10. 2019Learning by Abstraction: The Neural State Machinecited by 113
  11. 2020Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Textcited by 46
  12. 2019Semantic-Aware Scene Recognitioncited by 43
  13. 2020Are we pretraining it right? Digging deeper into visio-linguistic pretrainingcited by 25
  14. 2020Knowledge-Based Video Question Answering with Unsupervised Scene Descriptionscited by 15
  15. 2020Do Dogs have Whiskers? A New Knowledge Base of hasPart Relationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  2. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0