mlsanityData to 7 Oct 2026

M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

arXiv 2411.06176 ↗First posted 9 Nov 2024cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models. We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents. Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models. Our data, code, and models are available at https://multimodal-documents.github.io.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 9 Nov 2024.

Builds on

39 of 50 references.

References · 39

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
27 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  3. 2019ELI5: Long Form Question Answeringcited by 567
  4. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  9. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  10. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  12. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  13. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  14. 2022DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysiscited by 146
  15. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  16. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  17. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  18. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 59
  19. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  20. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  21. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  22. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  23. 2023PDFTriage: Question Answering over Long, Structured Documentscited by 29
  24. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  25. 2024LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understandingcited by 24
  26. 2024TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Datacited by 19
  27. 2023MarkQA: A large scale KBQA dataset with numerical reasoningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 21.

36 papers.

Sort:

By area:

  1. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  2. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  3. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
  4. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  5. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  6. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  7. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmarkcited by 6
  8. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  9. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  10. 2025Not All LoRA Parameters Are Essential: Insights on Inference Necessitycited by 4
  11. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understandingcited by 4
  12. 2026AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Constructioncited by 4
24 more
  1. 2026Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generationcited by 4
  2. 2025M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization?cited by 3
  3. 2025DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipelinecited by 3
  4. 2026Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QAcited by 2
  5. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understandingcited by 2
  6. 2026GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documentscited by 2
  7. 2025Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processingcited by 1
  8. 2026Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Modelscited by 1
  9. 2026Automatic Slide Updating with User-Defined Dynamic Templates and Natural Language Instructionscited by 1
  10. 2026From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAGcited by 1
  11. 2026Understanding the Behaviors of Environment-aware Information Retrievalcited by 1
  12. 2026CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisitioncited by 1
  13. 2025A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generationcited by 0
  14. 2026Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunkingcited by 0
  15. 2026ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstructioncited by 0
  16. 2026Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrievalcited by 0
  17. 2026DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmarkcited by 0
  18. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  19. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understandingcited by 0
  20. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  21. 2026XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understandingcited by 0
  22. 2026An Empirical Study of VLM Pipelines for Long-Document QAcited by 0
  23. 2026Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligencecited by 0
  24. 2026DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documents
  2. 2025Docopilot: Improving Multimodal Models for Document-Level Understanding
  3. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling
  4. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
  5. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  6. 2026MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation
  7. 2025WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts
  8. 2025URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding