mlsanityData to 7 Oct 2026

Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

arXiv 2109.04014 ↗First posted 9 Sep 2021cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

Knowledge-based visual question answering (VQA) requires answering questions with external knowledge in addition to the content of images. One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval. Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge. Because of varying knowledge bases, it is hard to fairly compare models' performance. To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge. We introduce various ways to retrieve knowledge using text and images and two reader styles: classification and extraction. Both the retriever and reader are trained with weak supervision. Our experimental results show that a good retriever can significantly improve the reader's performance on the OK-VQA challenge. The code and corpus are provided in https://github.com/luomancs/retrieverreaderforokvqa.git

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

33 of 59 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
21 more
  1. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  4. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  5. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  6. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  7. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  8. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  9. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  10. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  11. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  12. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  13. 2019Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answeringcited by 48
  14. 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48
  15. 2019Careful Selection of Knowledge to solve Open Book Question Answeringcited by 35
  16. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  17. 2020WeaQA: Weak Supervision via Captions for Visual Question Answeringcited by 25
  18. 2018Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answeringcited by 24
  19. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  20. 2018Adaptive Document Retrieval for Deep Question Answeringcited by 16
  21. 2021'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Taskscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 9; 2024: 17; 2025: 9; 2026: 3.

50 papers.

Sort:

By area:

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  2. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  3. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  4. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  5. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  6. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  7. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  8. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  9. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  10. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  11. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  12. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
38 more
  1. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  2. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  3. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  4. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  5. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  6. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  7. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  8. 2024A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysiscited by 14
  9. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  10. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  11. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  12. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 11
  13. 2023Combo of Thinking and Observing for Outside-Knowledge VQAcited by 9
  14. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answeringcited by 9
  15. 2024Rationale-guided Prompting for Knowledge-based Visual Question Answeringcited by 9
  16. 2024Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Modelscited by 8
  17. 2022Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 7
  18. 2024Multimodal Reranking for Knowledge-Intensive Visual Question Answeringcited by 6
  19. 2023Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answeringcited by 5
  20. 2024Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answeringcited by 5
  21. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5
  22. 2026InterLV-Search: Benchmarking Interleaved Multimodal Agentic Searchcited by 4
  23. 2024Knowledge Condensation and Reasoning for Knowledge-based VQAcited by 3
  24. 2024Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answeringcited by 3
  25. 2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
  26. 2024VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generationcited by 3
  27. 2024Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answeringcited by 3
  28. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  29. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  30. 2023Open-Set Knowledge-Based Visual Question Answering with Inference Pathscited by 2
  31. 2025Enabling Collaborative Parametric Knowledge Calibration for Retrieval-Augmented Vision Question Answeringcited by 2
  32. 2022Neural Retriever and Go Beyond: A Thesis Proposalcited by 1
  33. 2025From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answeringcited by 1
  34. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  35. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  36. 2025A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answeringcited by 0
  37. 2026Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answeringcited by 0
  38. 2026FashionKG-RAG: Knowledge Graph-Enhanced Retrieval-Augmented Generation for Fashion Question Answeringcited by 0