mlsanityData to 14 Oct 2026

Multi-Modal Answer Validation for Knowledge-Based VQA

arXiv 2103.12248 ↗First posted 23 Mar 2021cs.CV cs.CLcited by 79

Not on the map. Why?

Abstract (arXiv)

The problem of knowledge-based visual question answering involves answering questions that require external knowledge in addition to the content of the image. Such knowledge typically comes in various forms, including visual, textual, and commonsense knowledge. Using more knowledge sources increases the chance of retrieving more irrelevant or noisy facts, making it challenging to comprehend the facts and find the answer. To address this challenge, we propose Multi-modal Answer Validation using External knowledge (MAVEx), where the idea is to validate a set of promising answer candidates based on answer-specific knowledge retrieval. Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source. Our multi-modal setting is the first to leverage external visual knowledge (images searched using Google), in addition to textual knowledge in the form of Wikipedia sentences and ConceptNet concepts. Our experiments with OK-VQA, a challenging knowledge-based VQA dataset, demonstrate that MAVEx achieves new state-of-the-art results. Our code is available at https://github.com/jialinwu17/MAVEX

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

40 of 54 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017Mask R-CNNcited by 7,277
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
28 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  5. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  6. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  7. 2018Bilinear Attention Networkscited by 280
  8. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  9. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  10. 2017End-to-end Neural Coreference Resolutioncited by 241
  11. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 184
  12. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  13. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  14. 2014Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimationcited by 129
  15. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  16. 2021Decontextualization: Making Sentences Stand-Alonecited by 102
  17. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  18. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  19. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  20. 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
  21. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  22. 2015Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sourcescited by 42
  23. 2018Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answeringcited by 30
  24. 2017Answering Complex Questions Using Open Information Extractioncited by 25
  25. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  26. 2019Progressive Attention Memory Network for Movie Story Question Answeringcited by 21
  27. 2020Improving VQA and its Explanations by Comparing Competing Explanationscited by 12
  28. 2019Learning Rich Image Region Representation for Visual Question Answeringcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 15; 2023: 23; 2024: 18; 2025: 11; 2026: 5.

79 papers.

Sort:

By area:

  1. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  2. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  5. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  6. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  7. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  8. 2022Language Models are General-Purpose Interfacescited by 97
  9. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  10. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  11. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  12. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
38 more
  1. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  2. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 59
  3. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  4. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  5. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  6. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  7. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  8. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  9. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Modelscited by 30
  10. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  11. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  12. 2022VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answeringcited by 22
  13. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  14. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  15. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 19
  16. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQAcited by 18
  17. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  18. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  19. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  20. 2024Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answeringcited by 16
  21. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  22. 2023A Simple Baseline for Knowledge-Based Visual Question Answeringcited by 14
  23. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  24. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  25. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  26. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10
  27. 2023Combo of Thinking and Observing for Outside-Knowledge VQAcited by 9
  28. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answeringcited by 9
  29. 2024Rationale-guided Prompting for Knowledge-based Visual Question Answeringcited by 9
  30. 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injectioncited by 8
  31. 2024Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Modelscited by 8
  32. 2022Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 7
  33. 2024Multimodal Reranking for Knowledge-Intensive Visual Question Answeringcited by 6
  34. 2025Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inferencecited by 6
  35. 2023ArK: Augmented Reality with Knowledge Interactive Emergent Abilitycited by 5
  36. 2023Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Taskscited by 5
  37. 2023Benchmarking and Analyzing Generative Data for Visual Recognitioncited by 5
  38. 2024Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answeringcited by 5

50 most cited of 79 All 79 (JSON)