mlsanityData to 7 Oct 2026

Zero-shot Visual Question Answering using Knowledge Graph

arXiv 2107.05348 ↗First posted 12 Jul 2021cs.AIcited by 37

Not on the map. Why?

Abstract (arXiv)

Incorporating external knowledge to Visual Question Answering (VQA) has become a vital practical need. Existing methods mostly adopt pipeline approaches with different components for knowledge matching and extraction, feature learning, etc.However, such pipeline approaches suffer when some component does not perform well, which leads to error propagation and poor overall performance. Furthermore, the majority of existing approaches ignore the answer bias issue -- many answers may have never appeared during training (i.e., unseen answers) in real-word application. To bridge these gaps, in this paper, we propose a Zero-shot VQA algorithm using knowledge graphs and a mask-based learning mechanism for better incorporating external knowledge, and present new answer-based Zero-shot VQA splits for the F-VQA dataset. Experiments show that our method can achieve state-of-the-art performance in Zero-shot VQA with unseen answers, meanwhile dramatically augment existing end-to-end models on the normal F-VQA task.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

24 of 31 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  6. 2018Bilinear Attention Networkscited by 277
  7. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  8. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  9. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  10. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  11. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  12. 2019Commonsense Knowledge Base Completion with Structural and Semantic Contextcited by 61
12 more
  1. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  2. 2021OntoZSL: Ontology-enhanced Zero-shot Learningcited by 45
  3. 2015Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sourcescited by 42
  4. 2021Knowledge-aware Zero-Shot Learning: Survey and Perspectivecited by 41
  5. 2018Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answeringcited by 30
  6. 2019Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Modelscited by 28
  7. 2017C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Datasetcited by 24
  8. 2016Zero-Shot Visual Question Answeringcited by 20
  9. 2018Learning Answer Embeddings for Visual Question Answeringcited by 11
  10. 2018From Known to the Unknown: Transferring Knowledge to Answer Questions about Novel Visual and Semantic Conceptscited by 5
  11. 2017An Empirical Evaluation of Visual Question Answering for Novel Objectscited by 4
  12. 2020Visual Question Answering with Prior Class Semanticscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 7; 2023: 11; 2024: 10; 2025: 8.

37 papers.

Sort:

By area:

  1. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  2. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  3. 2022Exemplar Guided Deep Neural Network for Spatial Transcriptomics Analysis of Gene Expression Predictioncited by 39
  4. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
  5. 2022DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learningcited by 33
  6. 2022MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybridcited by 29
  7. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 28
  8. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  9. 2021Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Surveycited by 23
  10. 2023Context Disentangling and Prototype Inheriting for Robust Visual Groundingcited by 23
  11. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 20
  12. 2023Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignmentcited by 20
25 more
  1. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  2. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  3. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  4. 2024Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoningcited by 13
  5. 2024CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learningcited by 12
  6. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  7. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  8. 2022Disentangled Ontology Embedding for Zero-shot Learningcited by 8
  9. 2023Entity-Agnostic Representation Learning for Parameter-Efficient Knowledge Graph Embeddingcited by 8
  10. 2025ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answeringcited by 7
  11. 2024Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answeringcited by 5
  12. 2023FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answeringcited by 4
  13. 2025Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learningcited by 4
  14. 2024DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMscited by 3
  15. 2022Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graphcited by 2
  16. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  17. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Modelscited by 2
  18. 2023NeuralKG-ind: A Python Library for Inductive Knowledge Graph Representation Learningcited by 1
  19. 2024Intrinsic Subgraph Generation for Interpretable Graph based Visual Question Answeringcited by 1
  20. 2025Combining Knowledge Graph and LLMs for Enhanced Zero-shot Visual Question Answeringcited by 0
  21. 2025An Enhanced Large Language Model For Cross Modal Query Understanding System Using DL-KeyBERT Based CAZSSCL-MPGPTcited by 0
  22. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  23. 2025ViFP: A Framework for Visual False Positive Detection to Enhance Reasoning Reliability in VLMscited by 0
  24. 2025NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Taskscited by 0
  25. 2025Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Groundingcited by 0