mlsanityData to 7 Oct 2026

LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating

arXiv 2412.18424 ↗First posted 24 Dec 2024cs.AI cs.CLcited by 70

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large vision language models (LVLMs) have improved the document understanding capabilities remarkably, enabling the handling of complex document elements, longer contexts, and a wider range of tasks. However, existing document understanding benchmarks have been limited to handling only a small number of pages and fail to provide a comprehensive analysis of layout elements locating. In this paper, we first define three primary task categories: Long Document Understanding, numerical Reasoning, and cross-element Locating, and then propose a comprehensive benchmark, LongDocURL, integrating above three primary tasks and comprising 20 sub-tasks categorized based on different primary tasks and answer evidences. Furthermore, we develop a semi-automated construction pipeline and collect 2,325 high-quality question-answering pairs, covering more than 33,000 pages of documents, significantly outperforming existing benchmarks. Subsequently, we conduct comprehensive evaluation experiments on both open-source and closed-source models across 26 different configurations, revealing critical performance gaps in this field.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 29 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

24 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024GRAM: Global Reasoning for Multi-Page VQA7 Jan
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 21

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  5. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  7. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  8. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  9. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  10. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  11. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  12. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
9 more
  1. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  2. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  3. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  5. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  6. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  7. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  8. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequencescited by 12
  9. 2024Multi-Page Document Visual Question Answering using Self-Attention Scoring Mechanismcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 46.

70 papers.

Sort:

By area:

  1. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  2. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  3. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  4. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  5. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
  6. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  7. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
  8. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  9. 2025a1: Steep Test-time Scaling Law via Environment Augmented Generationcited by 13
  10. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  11. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  12. 2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
38 more
  1. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  2. 2025RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understandingcited by 8
  3. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  4. 2025Large Language Models as Computable Approximations to Solomonoff Inductioncited by 6
  5. 2025Do LLMs Truly Understand When a Precedent Is Overruled?cited by 6
  6. 2025URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understandingcited by 5
  7. 2026CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligencecited by 5
  8. 2025MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Taskscited by 4
  9. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understandingcited by 4
  10. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  11. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4
  12. 2026MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generationcited by 4
  13. 2026AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Constructioncited by 4
  14. 2026MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learningcited by 4
  15. 2026SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literaturecited by 3
  16. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 3
  17. 2025ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documentscited by 2
  18. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 2
  19. 2026BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selectioncited by 2
  20. 2026Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QAcited by 2
  21. 2026MAGE-RAG: Multigranular Adaptive Graph Evidence for Agentic Multimodal RAG in Long-Document QAcited by 2
  22. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understandingcited by 2
  23. 2026GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documentscited by 2
  24. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  25. 2026Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Languagecited by 1
  26. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextcited by 1
  27. 2026HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Documentcited by 1
  28. 2026AWM: Answerable Working Memory for Long-Document VQA Agentscited by 1
  29. 2026MUDDLE: Measuring Understanding of Documents under Distractor and Length Effectscited by 1
  30. 2025A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generationcited by 0
  31. 2026LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrievalcited by 0
  32. 2026MAB-DQA: Addressing Query Aspect Importance in Document Question Answering with Multi-Armed Banditscited by 0
  33. 2026MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsingcited by 0
  34. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  35. 2026Office Comprehension Benchmarkcited by 0
  36. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  37. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understandingcited by 0
  38. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
  2. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  3. 2025WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts
  4. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding
  5. 2025MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding
  6. 2026DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning
  7. 2026DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding
  8. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding