mlsanityData to 7 Oct 2026

Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines

arXiv 2502.16641 ↗First posted 23 Feb 2025cs.CV cs.CL cs.IRcited by 13

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. Current methods mainly employ separate retrieval and generation modules to acquire external knowledge and generate answers, respectively. We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task, which seamlessly integrates knowledge retriever into the generative multi-modal large language model, serving as a built-in search engine. Specifically, our model functions both as a generative retriever and an accurate answer generator. It not only helps retrieve documents from the knowledge base by producing identifiers for each document, but it also answers visual questions based on the retrieved documents. Furthermore, we propose a reinforced retrieval calibration module from relevance feedback to improve retrieval performance and align with the preferences for accurate answer generation. Extensive experiments on two representative OKVQA and A-OKVQA datasets demonstrate significant improvements ranging from 2.9% to 9.6% across all evaluation metrics when compared to strong baselines.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 47 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

32 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  2. 2022Retrieval Augmented Visual Question Answering with Outside Knowledge7 Oct
  3. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  12. 2022Transformer Memory as a Differentiable Search Indexcited by 242
17 more
  1. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  2. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  3. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  4. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  5. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  6. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  7. 2018Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoningcited by 51
  8. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  9. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  10. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  11. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyondcited by 27
  12. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 26
  13. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  14. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  15. 2023Combo of Thinking and Observing for Outside-Knowledge VQAcited by 9
  16. 2024From RAG to RICHES: Retrieval Interlaced with Sequence Generationcited by 5
  17. 2024Knowledge Condensation and Reasoning for Knowledge-based VQAcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025EventVAD: Training-Free Event-Aware Video Anomaly Detectioncited by 32
  2. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  3. 2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
  4. 2025TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistantcited by 6
  5. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  6. 2025Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answeringcited by 2
  7. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  8. 2025From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answeringcited by 1
  9. 2025Atom: Efficient On-Device Video-Language Pipelines Through Modular Reusecited by 0
  10. 2026SPARK-IL: Spectral Retrieval-Augmented RAG for Knowledge-driven Deepfake Detection via Incremental Learningcited by 0
  11. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  12. 2026From Clouds to Hallucinations: Atmospheric Retrieval Hijacking in Remote Sensing Vision-Language RAGcited by 0
1 more
  1. 2026Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  2. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework
  3. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  4. 2025QA-Dragon: Query-Aware Dynamic RAG System for Knowledge-Intensive Visual Question Answering
  5. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge
  6. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  7. 2025Enhancing Document VQA Models via Retrieval-Augmented Generation
  8. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries