# DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems

arXiv: https://arxiv.org/abs/2407.10701 · HTML: https://mlsanity.com/paper/2407.10701/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [OS-Copilot: Towards Generalist Computer Agents with Self-Improvement](https://mlsanity.com/paper/2402.07456/) (2024)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [Large Language Models in Law: A Survey](https://mlsanity.com/paper/2312.03718/) (2023)
- [doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset](https://mlsanity.com/paper/2011.06623/) (2020)
- [A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law](https://mlsanity.com/paper/2405.01769/) (2024)
- [LASER: LLM Agent with State-Space Exploration for Web Navigation](https://mlsanity.com/paper/2309.08172/) (2023)

## Built on by (most cited first, 30 of 37)

- [MMDocIR: Benchmarking Multimodal Retrieval for Long Documents](https://mlsanity.com/paper/2501.08828/) (2025)
- [Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training](https://mlsanity.com/paper/2508.00414/) (2025)
- [Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering](https://mlsanity.com/paper/2505.16470/) (2025)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [RAG-Anything: All-in-One RAG Framework](https://mlsanity.com/paper/2510.12323/) (2025)
- [MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs](https://mlsanity.com/paper/2507.20804/) (2025)
- [DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding](https://mlsanity.com/paper/2511.11552/) (2025)
- [PersoBench: Benchmarking Personalized Response Generation in Large Language Models](https://mlsanity.com/paper/2410.03198/) (2024)
- [A Survey on Human-Centric LLMs](https://mlsanity.com/paper/2411.14491/) (2024)
- [DocDancer: Towards Agentic Document-Grounded Information Seeking](https://mlsanity.com/paper/2601.05163/) (2026)
- [Cognitive Kernel: An Open-source Agent System towards Generalist Autopilots](https://mlsanity.com/paper/2409.10277/) (2024)
- [CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building](https://mlsanity.com/paper/2505.21069/) (2025)
- [T-GRAG: A Dynamic GraphRAG Framework for Resolving Temporal Conflicts and Redundancy in Knowledge Retrieval](https://mlsanity.com/paper/2508.01680/) (2025)
- [SCORE: A Semantic Evaluation Framework for Generative Document Parsing](https://mlsanity.com/paper/2509.19345/) (2025)
- [Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation](https://mlsanity.com/paper/2604.10741/) (2026)
- [Evaluating Hierarchical Clinical Document Classification Using Reasoning-Based LLMs](https://mlsanity.com/paper/2507.03001/) (2025)
- [SF-RAG: Structure-Fidelity Retrieval-Augmented Generation for Academic Question Answering](https://mlsanity.com/paper/2602.13647/) (2026)
- [MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA](https://mlsanity.com/paper/2606.05749/) (2026)
- [MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation](https://mlsanity.com/paper/2606.26458/) (2026)
- [GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents](https://mlsanity.com/paper/2607.11192/) (2026)
- [Human-AI Collaborative Game Testing with Vision Language Models](https://mlsanity.com/paper/2501.11782/) (2025)
- [Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processing](https://mlsanity.com/paper/2509.04469/) (2025)
- [BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.07329/) (2026)
- [DR3-Eval: Towards Realistic and Reproducible Deep Research Evaluation](https://mlsanity.com/paper/2604.14683/) (2026)
- [MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing](https://mlsanity.com/paper/2605.22100/) (2026)
- [DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations](https://mlsanity.com/paper/2607.19865/) (2026)
- [Comparing Credit Risk Estimates in the Gen-AI Era](https://mlsanity.com/paper/2506.07754/) (2025)
- [Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration](https://mlsanity.com/paper/2512.04445/) (2025)
- [Superlinear Multi-Step Attention](https://mlsanity.com/paper/2601.18401/) (2026)
- [Exploring Information Seeking Agent Consolidation](https://mlsanity.com/paper/2602.00585/) (2026)

## Abstract (arXiv)

Recently, there has been a growing interest among large language model (LLM) developers in LLM-based document reading systems, which enable users to upload their own documents and pose questions related to the document contents, going beyond simple reading comprehension tasks. Consequently, these systems have been carefully designed to tackle challenges such as file parsing, metadata extraction, multi-modal information understanding and long-context reading. However, no current benchmark exists to evaluate their performance in such scenarios, where a raw file and questions are provided as input, and a corresponding response is expected as output. In this paper, we introduce DocBench, a new benchmark designed to evaluate LLM-based document reading systems. Our benchmark involves a meticulously crafted process, including the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions. We evaluate both proprietary LLM-based systems accessible via web interfaces or APIs, and a parse-then-read pipeline employing open-source LLMs. Our evaluations reveal noticeable gaps between existing LLM-based document reading systems and human performance, underscoring the challenges of developing proficient systems. To summarize, DocBench aims to establish a standardized benchmark for evaluating LLM-based document reading systems under diverse real-world scenarios, thereby guiding future advancements in this research area.