# ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents

arXiv: https://arxiv.org/abs/2502.18017 · HTML: https://mlsanity.com/paper/2502.18017/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs](https://mlsanity.com/paper/2406.15319/) (2024)
- [MindSearch: Mimicking Human Minds Elicits Deep AI Searcher](https://mlsanity.com/paper/2407.20183/) (2024)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [Scaling Retrieval-Based Language Models with a Trillion-Token Datastore](https://mlsanity.com/paper/2407.12854/) (2024)
- [In Defense of RAG in the Era of Long-Context Language Models](https://mlsanity.com/paper/2409.01666/) (2024)

## Built on by (most cited first, 30 of 66)

- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2505.22019/) (2025)
- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.13186/) (2025)
- [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://mlsanity.com/paper/2601.08620/) (2026)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG](https://mlsanity.com/paper/2510.03663/) (2025)
- [AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10720/) (2025)
- [Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding](https://mlsanity.com/paper/2511.22850/) (2025)
- [GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts](https://mlsanity.com/paper/2601.05110/) (2026)
- [DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding](https://mlsanity.com/paper/2511.11552/) (2025)
- [LastingBench: Defend Benchmarks Against Knowledge Leakage](https://mlsanity.com/paper/2506.21614/) (2025)
- [Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning](https://mlsanity.com/paper/2508.02258/) (2025)
- [DualVLA: Building a Generalizable Embodied Agent via Partial Decoupling of Reasoning and Action](https://mlsanity.com/paper/2511.22134/) (2025)
- [Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models](https://mlsanity.com/paper/2510.01304/) (2025)
- [Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework](https://mlsanity.com/paper/2602.19549/) (2026)
- [Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding](https://mlsanity.com/paper/2510.15253/) (2025)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation](https://mlsanity.com/paper/2508.17079/) (2025)
- [Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?](https://mlsanity.com/paper/2508.03644/) (2025)
- [HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documents](https://mlsanity.com/paper/2511.20227/) (2025)
- [VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph](https://mlsanity.com/paper/2602.12735/) (2026)
- [AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.24134/) (2026)
- [Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations](https://mlsanity.com/paper/2603.01666/) (2026)
- [SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding](https://mlsanity.com/paper/2510.26615/) (2025)
- [Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agents](https://mlsanity.com/paper/2512.08870/) (2025)
- [CogDoc: Towards Unified thinking in Documents](https://mlsanity.com/paper/2512.12658/) (2025)
- [Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs](https://mlsanity.com/paper/2602.05275/) (2026)

## Abstract (arXiv)

Understanding information from visually rich documents remains a significant challenge for traditional Retrieval-Augmented Generation (RAG) methods. Existing benchmarks predominantly focus on image-based question answering (QA), overlooking the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. To bridge this gap, we introduce ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. Based on it, we identify key limitations in current RAG approaches: (i) purely visual retrieval methods struggle to effectively integrate both textual and visual features, and (ii) previous approaches often allocate insufficient reasoning tokens, limiting their effectiveness. To address these challenges, we propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy to effectively handle multi-modal retrieval. To further elicit the model's reasoning capabilities, we introduce an iterative agent workflow incorporating exploration, summarization, and reflection, providing a framework for investigating test-time scaling in RAG domains. Extensive experiments on ViDoSeek validate the effectiveness and generalization of our approach. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark. The code is available at https://github.com/Alibaba-NLP/ViDoRAG.