# ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

arXiv: https://arxiv.org/abs/2601.08620 · HTML: https://mlsanity.com/paper/2601.08620/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 22 of 22)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding](https://mlsanity.com/paper/2411.04952/) (2024)
- [BM25S: Orders of magnitude faster lexical search via eager sparse scoring](https://mlsanity.com/paper/2407.03618/) (2024)
- [Towards Complex Document Understanding By Discrete Reasoning](https://mlsanity.com/paper/2207.11871/) (2022)
- [ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents](https://mlsanity.com/paper/2502.18017/) (2025)
- [ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval](https://mlsanity.com/paper/2505.17166/) (2025)
- [REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark](https://mlsanity.com/paper/2502.12342/) (2025)
- [VISA: Retrieval Augmented Generation with Visual Source Attribution](https://mlsanity.com/paper/2412.14457/) (2024)
- [Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model](https://mlsanity.com/paper/2507.05513/) (2025)
- [ModernVBERT: Towards Smaller Visual Document Retrievers](https://mlsanity.com/paper/2510.01149/) (2025)
- [UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG](https://mlsanity.com/paper/2510.03663/) (2025)
- [FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents](https://mlsanity.com/paper/2504.13128/) (2025)
- [Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings](https://mlsanity.com/paper/2505.24782/) (2025)
- [Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report](https://mlsanity.com/paper/2510.14880/) (2025)

## Built on by (most cited first, 26 of 26)

- [Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrieval](https://mlsanity.com/paper/2602.03992/) (2026)
- [CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding](https://mlsanity.com/paper/2601.21262/) (2026)
- [AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction](https://mlsanity.com/paper/2603.29199/) (2026)
- [AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.24134/) (2026)
- [CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence](https://mlsanity.com/paper/2605.12882/) (2026)
- [BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence](https://mlsanity.com/paper/2603.07931/) (2026)
- [IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering](https://mlsanity.com/paper/2602.17687/) (2026)
- [NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval](https://mlsanity.com/paper/2603.12824/) (2026)
- [MINER: Mining Multimodal Internal Representation for Efficient Retrieval](https://mlsanity.com/paper/2605.06460/) (2026)
- [MinerU-Popo: Universal Post-Processing Model for Structured Document Parsing](https://mlsanity.com/paper/2605.24973/) (2026)
- [LEMUR: Learned Multi-Vector Retrieval](https://mlsanity.com/paper/2601.21853/) (2026)
- [AMES: Approximate Multi-modal Enterprise Search via Late Interaction Retrieval](https://mlsanity.com/paper/2603.13537/) (2026)
- [LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval](https://mlsanity.com/paper/2603.26683/) (2026)
- [Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model](https://mlsanity.com/paper/2603.28554/) (2026)
- [Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding](https://mlsanity.com/paper/2605.19866/) (2026)
- [ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAG](https://mlsanity.com/paper/2607.16208/) (2026)
- [KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval](https://mlsanity.com/paper/2608.01389/) (2026)
- [L"etzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents](https://mlsanity.com/paper/2608.21714/) (2026)
- [MCite-RL: Towards Reliable Multimodal RAG via Citation-enhanced Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.21808/) (2026)
- [ViSAR: Training-Free Adaptive-k Retrieval for Visual Document Question Answering](https://mlsanity.com/paper/2609.02486/) (2026)
- [ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport](https://mlsanity.com/paper/2609.34899/) (2026)
- [Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation](https://mlsanity.com/paper/2610.02148/) (2026)
- [TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning](https://mlsanity.com/paper/2610.09360/) (2026)
- [Simple Linear Corrections for Compression in Late Interaction](https://mlsanity.com/paper/2610.14196/) (2026)
- [CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets](https://mlsanity.com/paper/2610.15409/) (2026)
- [VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval](https://mlsanity.com/paper/2610.17057/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.