mlsanityData to 7 Oct 2026

REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark

arXiv 2502.12342 ↗First posted 17 Feb 2025cs.IR cs.CVcited by 34

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Accurate multi-modal document retrieval is crucial for Retrieval-Augmented Generation (RAG), yet existing benchmarks do not fully capture real-world challenges with their current design. We introduce REAL-MM-RAG, an automatically generated benchmark designed to address four key properties essential for real-world retrieval: (i) multi-modal documents, (ii) enhanced difficulty, (iii) Realistic-RAG queries and (iv) accurate labeling. Additionally, we propose a multi-difficulty-level scheme based on query rephrasing to evaluate models' semantic understanding beyond keyword matching. Our benchmark reveals significant model weaknesses, particularly in handling table-heavy documents and robustness to query rephrasing. To mitigate these shortcomings, we curate a rephrased training set and introduce a new finance-focused, table-heavy dataset. Fine-tuning on these datasets enables models to achieve state-of-the-art retrieval performance on REAL-MM-RAG benchmark. Our work offers a better way to evaluate and improve retrieval in multi-modal RAG systems while also providing training data and models that address current limitations.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 23 of its 38 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

23 of 38 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
11 more
  1. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  2. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  3. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  4. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  6. 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
  7. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 107
  8. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  9. 2021Evaluating the Robustness of Retrieval Pipelines with Query Variation Generatorscited by 33
  10. 2022Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellingscited by 23
  11. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 18.

34 papers.

Sort:

By area:

  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  2. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 22
  3. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  4. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  5. 2025DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankerscited by 9
  6. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  7. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  8. 2025Zero-shot Multimodal Document Retrieval via Cross-modal Question Generationcited by 6
  9. 2025MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generationcited by 6
  10. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  11. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  12. 2026Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generationcited by 4
22 more
  1. 2025SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understandingcited by 3
  2. 2025CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolutioncited by 3
  3. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  4. 2025Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrievalcited by 2
  5. 2025MuISQA: Multi-Intent Retrieval-Augmented Generation for Scientific Question Answeringcited by 2
  6. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 2
  7. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  8. 2026MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generationcited by 2
  9. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  10. 2025Optimizing Retrieval for RAG via Reinforcement Learningcited by 1
  11. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
  12. 2026Document-as-Image Representations Fall Short for Scientific Retrievalcited by 1
  13. 2026From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAGcited by 1
  14. 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrievalcited by 1
  15. 2026Little Brains, Big Feats: Exploring Compact Language Modelscited by 0
  16. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  17. 2026KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrievalcited by 0
  18. 2026What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answeringcited by 0
  19. 2026Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinementcited by 0
  20. 2026KoViDoRe: Korean Visual Document Retrievalcited by 0
  21. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelinescited by 0
  22. 2026MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  2. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts
  3. 2026From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents
  4. 2024Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines
  5. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  6. 2025Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning
  7. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
  8. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation