mlsanityData to 14 Oct 2026

KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA

arXiv 2012.11014 ↗First posted 20 Dec 2020cs.CV cs.CLcited by 125

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pre-training and supervised training data with transformer-based models. Second, explicit, symbolic knowledge encoded in knowledge bases. Our approach combines both - exploiting the powerful implicit reasoning of transformer models for answer prediction, and integrating symbolic representations from a knowledge graph, while never losing their explicit semantics to an implicit embedding. We combine diverse sources of knowledge to cover the wide variety of knowledge needed to solve knowledge-based questions. We show our approach, KRISP (Knowledge Reasoning with Implicit and Symbolic rePresentations), significantly outperforms state-of-the-art on OK-VQA, the largest available dataset for open-domain knowledge-based VQA. We show with extensive ablations that while our model successfully exploits implicit knowledge reasoning, the symbolic answer module which explicitly connects the knowledge graph to the answer vocabulary is critical to the performance of our method and generalizes to rare answers.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 32; 2023: 35; 2024: 32; 2025: 13; 2026: 5.

125 papers.

Sort:

By area:

  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  4. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  5. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  6. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  8. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  9. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  10. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  11. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  12. 2022Language Models are General-Purpose Interfacescited by 97
38 more
  1. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  2. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  3. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  4. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  6. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  7. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoningcited by 60
  8. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 59
  9. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  10. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  11. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  12. 2021Incorporating Domain Knowledge into Deep Neural Networkscited by 56
  13. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  14. 2023EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answeringcited by 52
  15. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  16. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  17. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  18. 2022Neural Approaches to Conversational Information Retrievalcited by 42
  19. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  20. 2022MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answeringcited by 36
  21. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  22. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 35
  23. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Modelscited by 30
  24. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  25. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  26. 2021Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Surveycited by 23
  27. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  28. 2022VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answeringcited by 22
  29. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  30. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  31. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  32. 2023Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!cited by 19
  33. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 19
  34. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learningcited by 19
  35. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQAcited by 18
  36. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  37. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 16
  38. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16

50 most cited of 125 All 125 (JSON)