mlsanityData to 7 Oct 2026

Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries

arXiv 2502.16636 ↗First posted 23 Feb 2025cs.CL cs.CVcited by 16

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) is a paradigm that augments large language models (LLMs) with external knowledge to tackle knowledge-intensive question answering. While several benchmarks evaluate Multimodal LLMs (MLLMs) under Multimodal RAG settings, they predominantly retrieve from textual corpora and do not explicitly assess how models exploit visual evidence during generation. Consequently, there still lacks benchmark that isolates and measures the contribution of retrieved images in RAG. We introduce Visual-RAG, a question-answering benchmark that targets visually grounded, knowledge-intensive questions. Unlike prior work, Visual-RAG requires text-to-image retrieval and the integration of retrieved clue images to extract visual evidence for answer generation. With Visual-RAG, we evaluate 5 open-source and 3 proprietary MLLMs, showcasing that images provide strong evidence in augmented generation. However, even state-of-the-art models struggle to efficiently extract and utilize visual knowledge. Our results highlight the need for improved visual retrieval, grounding, and attribution in multimodal RAG systems.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 54 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

31 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation14 Dec
  2. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov
  3. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models10 Oct
  4. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs28 Jun
  5. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories15 Jun

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
14 more
  1. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  2. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  3. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  4. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  5. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  6. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  7. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  8. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  9. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  10. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  11. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  12. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  13. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  14. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  2. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  3. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  4. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  5. 2025Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitationcited by 6
  6. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  7. 2026Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learningcited by 4
  8. 2025VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrievalcited by 3
  9. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  10. 2026Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generationcited by 1
  11. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Modelscited by 1
  12. 2025ADSeeker: A Knowledge-Grounded Reasoning Framework for Industry Anomaly Detection and Reasoningcited by 0
4 more
  1. 2025Embedding the Teacher: Distilling vLLM Preferences for Scalable Image Retrievalcited by 0
  2. 2026PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Vision-Language Modelscited by 0
  3. 2026Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generationcited by 0
  4. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  2. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
  3. 2026KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
  4. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  5. 2024Searching for Best Practices in Retrieval-Augmented Generation
  6. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
  7. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  8. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA