mlsanityData to 7 Oct 2026

VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation

arXiv 2412.10704 ↗First posted 14 Dec 2024cs.CLcited by 40

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings with rich multimodal content, including tables, charts, and presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG, combining robust visual retrieval capabilities with sophisticated linguistic reasoning. VisDoMRAG employs a multi-step reasoning process encompassing evidence curation and chain-of-thought reasoning for concurrent textual and visual RAG pipelines. A key novelty of VisDoMRAG is its consistency-constrained modality fusion mechanism, which aligns the reasoning processes across modalities at inference time to produce a coherent final answer. This leads to enhanced accuracy in scenarios where critical information is distributed across modalities and improved answer verifiability through implicit context attribution. Through extensive experiments involving open-source and proprietary large language models, we benchmark state-of-the-art document QA methods on VisDoMBench. Extensive results show that VisDoMRAG outperforms unimodal and long-context LLM baselines for end-to-end multimodal document QA by 12-20%.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 55 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

32 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Retrieval Augmented Visual Question Answering with Outside Knowledge7 Oct
  2. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 30

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  10. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
18 more
  1. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  4. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  5. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  6. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  7. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  8. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  9. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  10. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  11. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  12. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  13. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  14. 2023MultiTabQA: Generating Tabular Answers for Multi-Table Question Answeringcited by 32
  15. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  16. 2024The Ethics of Interaction: Mitigating Security Threats in LLMscited by 30
  17. 2019Text Generation with Exemplar-based Adaptive Decodingcited by 28
  18. 2022V-Doc : Visual questions answers with Documentscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 23.

40 papers.

Sort:

By area:

  1. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  3. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  4. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  5. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  6. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
  7. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
  8. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  9. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  10. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  11. 2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
  12. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
28 more
  1. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  2. 2025DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collectionscited by 7
  3. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  4. 2026MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learningcited by 4
  5. 2026Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchainscited by 3
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  7. 2025How does Watermarking Affect Visual Language Models in Document Understanding?cited by 1
  8. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
  9. 2026G2-Reader: Dual Evolving Graphs for Multimodal Document QAcited by 1
  10. 2026RCP: Representation Consistency Pruner for Mitigating Distribution Shift in Large Vision-Language Modelscited by 1
  11. 2026FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&Acited by 1
  12. 2026HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Documentcited by 1
  13. 2026Citation-Enforced RAG for Fiscal Document Intelligence: Cited, Explainable Knowledge Retrieval in Tax Compliancecited by 0
  14. 2026LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrievalcited by 0
  15. 2026AdaQE-CG: Adaptive Query Expansion for Web-Scale Generative AI Model and Data Card Generationcited by 0
  16. 2026LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understandingcited by 0
  17. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  18. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  19. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  20. 2026Reason Before You Retrieve: Agentic Planning for Multi-modal RAGcited by 0
  21. 2026VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documentscited by 0
  22. 2026Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attributioncited by 0
  23. 2026D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understandingcited by 0
  24. 2026HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answeringcited by 0
  25. 2026ViSAR: Training-Free Adaptive-k Retrieval for Visual Document Question Answeringcited by 0
  26. 2026An Empirical Study of VLM Pipelines for Long-Document QAcited by 0
  27. 2026Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligencecited by 0
  28. 2026TAEC: Trajectory-Aware Evidence Coordination for Multi-Step Visual RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering
  2. 2026AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction
  3. 2026MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
  4. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  5. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understanding
  6. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
  7. 2025HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documents
  8. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding