mlsanityData to 14 Oct 2026

Docopilot: Improving Multimodal Models for Document-Level Understanding

arXiv 2507.14675 ↗First posted 19 Jul 2025cs.CV cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level datasets. While current retrieval-augmented generation (RAG) methods offer partial solutions, they suffer from issues, such as fragmented retrieval contexts, multi-stage error accumulation, and extra time costs of retrieval. In this work, we present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents. This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents. Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG. Experiments demonstrate that Docopilot achieves superior coherence, accuracy, and efficiency in document understanding tasks and multi-turn interactions, setting a new baseline for document-level multimodal understanding. Data, code, and models are released at https://github.com/OpenGVLab/Docopilot

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 96 references that are in the map. First posted on arXiv 19 Jul 2025.

Builds on

82 of 96 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ColPali: Efficient Document Retrieval with Vision Language Models27 Jun
  2. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models17 Jun
  3. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding8 Apr
  4. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  5. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding20 Nov
  6. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul

Other references · 76

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  13. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  15. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  16. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  17. 2017Learning multiple visual domains with residual adapterscited by 557
  18. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  19. 2021InfographicVQAcited by 437
  20. 2021OCR-free Document Understanding Transformercited by 407
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  22. 2024InternLM2 Technical Reportcited by 397
  23. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  24. 2019Scene Text Visual Question Answeringcited by 331
  25. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  26. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  27. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  28. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  29. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  30. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  31. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  32. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  33. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  34. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  35. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  36. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  37. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  38. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  2. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  3. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  4. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  5. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  6. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
  7. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4
  8. 2025CogDoc: Towards Unified thinking in Documentscited by 3
  9. 2025HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discoverycited by 3
  10. 2025MedRepBench: A Comprehensive Benchmark for Medical Report Interpretationcited by 2
  11. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understandingcited by 2
  12. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
7 more
  1. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 2
  2. 2026MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QAcited by 2
  3. 2026AWM: Answerable Working Memory for Long-Document VQA Agentscited by 2
  4. 2026MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reportscited by 1
  5. 2026BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehensioncited by 0
  6. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generationcited by 0
  7. 2026Block-Sparse Attention with Semantic-Geometric Decoupled Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  2. 2025Zero-Shot Document Understanding using Pseudo Table of Contents-Guided Retrieval-Augmented Generation
  3. 2024Read and Think: An Efficient Step-wise Multimodal Language Model for Document Understanding and Reasoning
  4. 2026Multimodal Graph RAG for Long-range Visually Rich Document Understanding
  5. 2026LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding
  6. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
  7. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  8. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding