mlsanityData to 7 Oct 2026

MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding

arXiv 2503.13964 ↗First posted 18 Mar 2025cs.LGcited by 66

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Document Question Answering (DocQA) is a very common task. Existing methods using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and Retrieval Augmented Generation (RAG) often prioritize information from a single modal, failing to effectively integrate textual and visual cues. These approaches struggle with complex multi-modal reasoning, limiting their performance on real-world documents. We present MDocAgent (A Multi-Modal Multi-Agent Framework for Document Understanding), a novel RAG and multi-agent framework that leverages both text and image. Our system employs five specialized agents: a general agent, a critical agent, a text agent, an image agent and a summarizing agent. These agents engage in multi-modal context retrieval, combining their individual insights to achieve a more comprehensive understanding of the document's content. This collaborative approach enables the system to synthesize information from both textual and visual components, leading to improved accuracy in question answering. Preliminary experiments on five benchmarks like MMLongBench, LongDocURL demonstrate the effectiveness of our MDocAgent, achieve an average improvement of 12.1% compared to current state-of-the-art method. This work contributes to the development of more robust and comprehensive DocQA systems capable of handling the complexities of real-world documents containing rich textual and visual information. Our data and code are available at https://github.com/aiming-lab/MDocAgent.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 49 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

46 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling24 Feb

Other references · 45

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
33 more
  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  7. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  8. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  9. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  10. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  11. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  12. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  13. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  14. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  15. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  16. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  17. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  18. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  19. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  20. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  21. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  22. 2020Adapting Object Detectors with Conditional Domain Normalizationcited by 55
  23. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  24. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  25. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  26. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  27. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  28. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  29. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  30. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  31. 2022V-Doc : Visual questions answers with Documentscited by 19
  32. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  33. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understandingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 39.

66 papers.

Sort:

By area:

  1. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  2. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  5. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  6. 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
  7. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  8. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
  9. 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 25
  10. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  11. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  12. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
38 more
  1. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  2. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  3. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  4. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  5. 2025A Multimodal Multi-Agent Framework for Radiology Report Generationcited by 8
  6. 2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
  7. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  8. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  9. 2025Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Railscited by 7
  10. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  11. 2025From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualizationcited by 5
  12. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  13. 2026AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generationcited by 5
  14. 2026MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learningcited by 4
  15. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  16. 2025SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understandingcited by 3
  17. 2026FileGram: Grounding Agent Personalization in File-System Behavioral Tracescited by 3
  18. 2026PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMscited by 3
  19. 2026AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoningcited by 3
  20. 2025ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documentscited by 2
  21. 2025ARIAL: An Agentic Framework for Document VQA with Precise Answer Localizationcited by 2
  22. 2026SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generationcited by 2
  23. 2026DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understandingcited by 2
  24. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  25. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understandingcited by 2
  26. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  27. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  28. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  29. 2025FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computationcited by 1
  30. 2026BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generationcited by 1
  31. 2026SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Readcited by 1
  32. 2026Document-as-Image Representations Fall Short for Scientific Retrievalcited by 1
  33. 2026AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelismcited by 1
  34. 2026Hierarchical Attacks for Multi-Modal Multi-Agent Reasoningcited by 1
  35. 2026MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QAcited by 1
  36. 2026Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesiscited by 1
  37. 2026Exploring Information Seeking Agent Consolidationcited by 0
  38. 2026DeALOG: Decentralized Multi-Agents Log-Mediated Reasoning Frameworkcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
  2. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understanding
  3. 2026ICDAR2026 Competition on Multimodal Reasoning over Documents in Multiple Domains
  4. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents
  5. 2026ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering
  6. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
  7. 2025MHier-RAG: Multi-Modal RAG for Visual-Rich Document Question-Answering via Hierarchical and Multi-Granularity Reasoning
  8. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answering