mlsanityData to 7 Oct 2026

Retrieval Augmented Visual Question Answering with Outside Knowledge

arXiv 2210.03809 ↗First posted 7 Oct 2022cs.CLcited by 93

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance. Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 62 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

38 of 62 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
26 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  4. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  6. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  9. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  10. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  11. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  12. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  13. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  14. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  15. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  16. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  17. 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
  18. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  19. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  20. 2019Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answeringcited by 68
  21. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  22. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  23. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  24. 2019Bilinear Graph Networks for Visual Question Answeringcited by 22
  25. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  26. 2021End-to-End Entity Resolution and Question Answering Using Differentiable Knowledge Graphscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 25; 2025: 29; 2026: 23.

93 papers.

Sort:

By area:

  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  4. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  5. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  6. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  7. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  8. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  9. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  10. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  11. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  12. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 38
38 more
  1. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applicationscited by 31
  2. 2024On the Evaluation of Machine-Generated Reportscited by 23
  3. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  4. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 18
  5. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  6. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  7. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  8. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  9. 2024A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysiscited by 14
  10. 2023Semi-Parametric Video-Grounded Text Generationcited by 13
  11. 2024EdgeRAG: Online-Indexed RAG for Edge Devicescited by 13
  12. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  13. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  14. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  15. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  16. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  17. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 11
  18. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  19. 2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
  20. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answeringcited by 9
  21. 2024Synthetic Multimodal Question Generationcited by 8
  22. 2024Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Modelscited by 8
  23. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  24. 2023Survey on Memory-Augmented Neural Networks: Cognitive Insights to AI Applicationscited by 7
  25. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  26. 2025Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inferencecited by 6
  27. 2026SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glassescited by 6
  28. 2024ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusioncited by 5
  29. 2024Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answeringcited by 5
  30. 2025Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generationcited by 5
  31. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  32. 2023FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answeringcited by 4
  33. 2024Few-Shot VQA with Frozen LLMs: A Tale of Two Approachescited by 4
  34. 2024Self-adaptive Multimodal Retrieval-Augmented Generationcited by 4
  35. 2025CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planningcited by 4
  36. 2024Knowledge Condensation and Reasoning for Knowledge-based VQAcited by 3
  37. 2024Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answeringcited by 3
  38. 2024VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generationcited by 3

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA
  2. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering
  3. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
  5. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  6. 2024Find The Gap: Knowledge Base Reasoning For Visual Question Answering
  7. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQA
  8. 2025Enhancing Document VQA Models via Retrieval-Augmented Generation