mlsanityData to 7 Oct 2026

Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

arXiv 2411.02937 ↗First posted 5 Nov 2024cs.CLcited by 65

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 2 more
Abstract (arXiv)

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 53 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

45 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
32 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  9. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  10. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  11. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  12. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  13. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  14. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  15. 2023Multimodal Large Language Models: A Surveycited by 193
  16. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  17. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  18. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  19. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  20. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  21. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  22. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  23. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  24. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoningcited by 46
  25. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  26. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  27. 2024McEval: Massively Multilingual Code Evaluationcited by 42
  28. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  29. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  30. 2024SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligencecited by 20
  31. 2024II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answeringcited by 4
  32. 2024Raw Text is All you Need: Knowledge-intensive Multi-turn Instruction Tuning for Large Language Modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 41; 2026: 23.

65 papers.

Sort:

By area:

  1. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  4. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  5. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  6. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  7. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  8. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  9. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  10. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  11. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  12. 2025DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokenscited by 21
38 more
  1. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Frameworkcited by 18
  2. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  3. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9
  4. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluationcited by 8
  5. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  6. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  7. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  8. 2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agentscited by 7
  9. 2025Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inferencecited by 6
  10. 2025Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitationcited by 6
  11. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Strongercited by 6
  12. 2026SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glassescited by 6
  13. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learningcited by 6
  14. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  15. 2025WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generationcited by 5
  16. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  17. 2026BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agentscited by 5
  18. 2025Semi-supervised Node Importance Estimation with Informative Distribution Modeling for Uncertainty Regularizationcited by 4
  19. 2025From Token to Line: Enhancing Code Generation with a Long-Term Perspectivecited by 4
  20. 2025Neuro-Symbolic Query Compilercited by 4
  21. 2025Taming the Untamed: Graph-Based Knowledge Retrieval and Reasoning for MLLMs to Conquer the Unknowncited by 4
  22. 2025MDIT: A Model-free Data Interpolation Method for Diverse Instruction Tuningcited by 3
  23. 2025Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMscited by 3
  24. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  25. 2026M3Searcher: Modular Multimodal Information Seeking Agency with Retrieval-Oriented Reasoningcited by 3
  26. 2026Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learningcited by 3
  27. 2026GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Tracescited by 3
  28. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2
  29. 2025CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generationcited by 2
  30. 2025MMhops-R1: Multimodal Multi-hop Reasoningcited by 2
  31. 2026Efficient Multimodal Planning Agent for Visual Question-Answeringcited by 2
  32. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  33. 2026MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generationcited by 2
  34. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  35. 2025MCiteBench: A Multimodal Benchmark for Generating Text with Citationscited by 1
  36. 2025OnRL-RAG: Real-Time Personalized Mental Health Dialogue Systemcited by 1
  37. 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuningcited by 1
  38. 2025Efficient Agent: Optimizing Planning Capability for Multimodal Retrieval Augmented Generationcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
  2. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  3. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
  4. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
  5. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
  6. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  7. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA
  8. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering