mlsanityData to 7 Oct 2026

mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA

arXiv 2411.15041 ↗First posted 22 Nov 2024cs.AI cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Advanced Multimodal Large Language Models (MLLMs) struggle with recent Knowledge-based Visual Question Answering (VQA) tasks, such as INFOSEEK and Encyclopedic-VQA, due to their limited and frozen knowledge scope, often leading to ambiguous and inaccurate responses. Thus, multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge, effectively expanding the knowledge scope. However, current mRAG methods have inherent drawbacks, including: 1) Performing retrieval even when external knowledge is not needed. 2) Lacking of identification of evidence that supports the query. 3) Increasing model complexity due to additional information filtering modules or rules. To address these shortcomings, we propose a novel generalized framework called multimodal Retrieval-Reflection-Augmented Generation (mR2AG), which achieves adaptive retrieval and useful information localization to enable answers through two easy-to-implement reflection operations, preventing high model complexity. In mR2AG, Retrieval-Reflection is designed to distinguish different user queries and avoids redundant retrieval calls, and Relevance-Reflection is introduced to guide the MLLM in locating beneficial evidence of the retrieved content and generating answers accordingly. In addition, mR2AG can be integrated into any well-trained MLLM with efficient fine-tuning on the proposed mR2AG Instruction-Tuning dataset (mR2AG-IT). mR2AG significantly outperforms state-of-the-art MLLMs (e.g., GPT-4o) and mRAG-based MLLMs on INFOSEEK and Encyclopedic-VQA, while maintaining the exceptional capabilities of base MLLMs across a wide range of Visual-dependent tasks.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 69 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

46 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models11 Oct
  2. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul
  3. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
31 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  17. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  18. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  19. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  20. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  21. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  22. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  23. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  24. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  25. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  26. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  27. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Modelscited by 24
  28. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  29. 2024HAMMR: HierArchical MultiModal React agents for generic VQAcited by 7
  30. 2023How (not) to ensemble LVLMs for VQAcited by 4
  31. 2023LOIS: Looking Out of Instance Semantics for Visual Question Answeringcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 10
  4. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answeringcited by 7
  5. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  6. 2025CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planningcited by 4
  7. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  8. 2026CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoningcited by 3
  9. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  10. 2025MMhops-R1: Multimodal Multi-hop Reasoningcited by 2
  11. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  12. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
7 more
  1. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMscited by 1
  2. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
  3. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answeringcited by 1
  4. 2025M3Prune: Hierarchical Communication Graph Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generationcited by 0
  5. 2026MMAgent-R2: Learning to Rerank and Reject for Agentic mRAGcited by 0
  6. 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAGcited by 0
  7. 2026UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
  2. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
  3. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  4. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  5. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
  6. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
  7. 2026Efficient Multimodal Planning Agent for Visual Question-Answering
  8. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text