mlsanityData to 7 Oct 2026

MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations

arXiv 2407.01523 ↗First posted 1 Jul 2024cs.CV cs.CLcited by 174

Not on the map. Why?

Cited by 51 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding documents with rich layouts and multi-modal components is a long-standing and practical task. Recent Large Vision-Language Models (LVLMs) have made remarkable strides in various tasks, particularly in single-page document understanding (DU). However, their abilities on long-context DU remain an open problem. This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. Distinct from previous datasets, it is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Towards comprehensive evaluation, answers to these questions rely on pieces of evidence from (1) different sources (text, image, chart, table, and layout structure) and (2) various locations (i.e. page number). Moreover, 33.2% of the questions are cross-page questions requiring evidence across multiple pages. 22.8% of the questions are designed to be unanswerable for detecting potential hallucinations. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models. Notably, the best-performing model, GPT-4o, achieves an F1 score of only 42.7%, while the second-best, GPT-4V, scores 31.4%. Furthermore, 12 LVLMs (all except GPT-4o and GPT-4V) even present worse performance than their LLM counterparts which are fed with lossy-parsed OCR documents. These results validate the necessity of future research toward more capable long-context LVLMs. Project Page: https://mayubo2333.github.io/MMLongBench-Doc

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

40 of 64 references.

References · 40

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  11. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  12. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
28 more
  1. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  2. 2021InfographicVQAcited by 430
  3. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  4. 2021OCR-free Document Understanding Transformercited by 400
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  8. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  9. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  10. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  11. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  13. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  14. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  15. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  16. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  17. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  18. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  19. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  20. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  21. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  22. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  23. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  24. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  25. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  26. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  27. 2014Growth rates of modern science: A bibliometric analysis based on the number of publications and cited referencescited by 63
  28. 2020AxCell: Automatic Extraction of Results from Machine Learning Paperscited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 59; 2026: 102.

174 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  3. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  4. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  6. 2026Qwen3.5-Omni Technical Reportcited by 146
  7. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  8. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  9. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  10. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  11. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  12. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 64
38 more
  1. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  3. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  4. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  5. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  6. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  7. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  8. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  9. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 34
  10. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  11. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
  12. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  13. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  14. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  15. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  16. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  17. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  18. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  19. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  20. 2025RAG-Anything: All-in-One RAG Frameworkcited by 18
  21. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  22. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
  23. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  24. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  25. 2025BookRAG: A Hierarchical Structure-aware Index-based Approach for Retrieval-Augmented Generation on Complex Documentscited by 16
  26. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  27. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  28. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  29. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  30. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  31. 2025WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Chartscited by 8
  32. 2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
  33. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  34. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  35. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  36. 2026EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agentscited by 7
  37. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?cited by 6
  38. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  2. 2026How to Train Your Long-Context Visual Document Model
  3. 2026BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models
  4. 2024MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysis
  5. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  6. 2025Docopilot: Improving Multimodal Models for Document-Level Understanding
  7. 2026DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
  8. 2025MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding