# Docopilot: Improving Multimodal Models for Document-Level Understanding

arXiv: https://arxiv.org/abs/2507.14675 · HTML: https://mlsanity.com/paper/2507.14675/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 82 of its 96 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 30 of 82)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)

## Built on by (most cited first, 19 of 19)

- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding](https://mlsanity.com/paper/2511.22850/) (2025)
- [Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding](https://mlsanity.com/paper/2510.15253/) (2025)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)
- [DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting](https://mlsanity.com/paper/2602.15958/) (2026)
- [RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension](https://mlsanity.com/paper/2601.14289/) (2026)
- [CogDoc: Towards Unified thinking in Documents](https://mlsanity.com/paper/2512.12658/) (2025)
- [HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery](https://mlsanity.com/paper/2512.22899/) (2025)
- [MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation](https://mlsanity.com/paper/2508.16674/) (2025)
- [DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding](https://mlsanity.com/paper/2511.11313/) (2025)
- [Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey](https://mlsanity.com/paper/2601.03262/) (2025)
- [See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding](https://mlsanity.com/paper/2605.18018/) (2026)
- [MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA](https://mlsanity.com/paper/2606.05749/) (2026)
- [AWM: Answerable Working Memory for Long-Document VQA Agents](https://mlsanity.com/paper/2608.25618/) (2026)
- [MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports](https://mlsanity.com/paper/2605.03103/) (2026)
- [BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension](https://mlsanity.com/paper/2607.05614/) (2026)
- [Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.30163/) (2026)
- [Block-Sparse Attention with Semantic-Geometric Decoupled Routing](https://mlsanity.com/paper/2609.22884/) (2026)

## Abstract (arXiv)

Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level datasets. While current retrieval-augmented generation (RAG) methods offer partial solutions, they suffer from issues, such as fragmented retrieval contexts, multi-stage error accumulation, and extra time costs of retrieval. In this work, we present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents. This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents. Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG. Experiments demonstrate that Docopilot achieves superior coherence, accuracy, and efficiency in document understanding tasks and multi-turn interactions, setting a new baseline for document-level multimodal understanding. Data, code, and models are released at https://github.com/OpenGVLab/Docopilot