# M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

arXiv: https://arxiv.org/abs/2411.06176 · HTML: https://mlsanity.com/paper/2411.06176/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 9 Nov 2024.

## Builds on (most cited first, 30 of 39)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [RAFT: Adapting Language Model to Domain Specific RAG](https://mlsanity.com/paper/2403.10131/) (2024)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [ColPali: Efficient Document Retrieval with Vision Language Models](https://mlsanity.com/paper/2407.01449/) (2024)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?](https://mlsanity.com/paper/2404.05955/) (2024)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)

## Built on by (most cited first, 30 of 36)

- [LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating](https://mlsanity.com/paper/2412.18424/) (2024)
- [Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering](https://mlsanity.com/paper/2505.16470/) (2025)
- [SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement](https://mlsanity.com/paper/2506.14035/) (2025)
- [Scaling Language-Centric Omnimodal Representation Learning](https://mlsanity.com/paper/2510.11693/) (2025)
- [Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding](https://mlsanity.com/paper/2511.22850/) (2025)
- [VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding](https://mlsanity.com/paper/2508.07493/) (2025)
- [Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark](https://mlsanity.com/paper/2507.15882/) (2025)
- [AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction](https://mlsanity.com/paper/2603.29199/) (2026)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends](https://mlsanity.com/paper/2501.02235/) (2025)
- [Not All LoRA Parameters Are Essential: Insights on Inference Necessity](https://mlsanity.com/paper/2503.23360/) (2025)
- [Multi-Agent Interactive Question Generation Framework for Long Document Understanding](https://mlsanity.com/paper/2507.20145/) (2025)
- [Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation](https://mlsanity.com/paper/2604.10741/) (2026)
- [M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization?](https://mlsanity.com/paper/2503.21839/) (2025)
- [DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline](https://mlsanity.com/paper/2504.14145/) (2025)
- [Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA](https://mlsanity.com/paper/2605.14928/) (2026)
- [Hierarchical Evidence-Driven Reasoning for Long Document Understanding](https://mlsanity.com/paper/2607.04625/) (2026)
- [GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents](https://mlsanity.com/paper/2607.11192/) (2026)
- [Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processing](https://mlsanity.com/paper/2509.04469/) (2025)
- [Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2601.03926/) (2026)
- [Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking](https://mlsanity.com/paper/2602.21196/) (2026)
- [Automatic Slide Updating with User-Defined Dynamic Templates and Natural Language Instructions](https://mlsanity.com/paper/2604.17894/) (2026)
- [From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG](https://mlsanity.com/paper/2605.15019/) (2026)
- [Understanding the Behaviors of Environment-aware Information Retrieval](https://mlsanity.com/paper/2606.16817/) (2026)
- [CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition](https://mlsanity.com/paper/2607.25294/) (2026)
- [Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence](https://mlsanity.com/paper/2609.32041/) (2026)
- [A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation](https://mlsanity.com/paper/2509.02864/) (2025)
- [ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction](https://mlsanity.com/paper/2604.23813/) (2026)
- [Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval](https://mlsanity.com/paper/2605.24530/) (2026)
- [DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark](https://mlsanity.com/paper/2605.30027/) (2026)

## Abstract (arXiv)

The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models. We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents. Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models. Our data, code, and models are available at https://multimodal-documents.github.io.