# A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents

arXiv: https://arxiv.org/abs/2303.09957 · HTML: https://mlsanity.com/paper/2303.09957/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2023.

## Built on by (most cited first, 11 of 11)

- [MedDM:LLM-executable clinical guidance tree for clinical decision-making](https://mlsanity.com/paper/2312.02441/) (2023)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources](https://mlsanity.com/paper/2405.01930/) (2024)
- [MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs](https://mlsanity.com/paper/2505.19800/) (2025)
- [Unsupervised Domain Adaptation for Keyphrase Generation using Citation Contexts](https://mlsanity.com/paper/2409.13266/) (2024)
- [Can Large Language Models Infer Causal Relationships from Real-World Text?](https://mlsanity.com/paper/2505.18931/) (2025)
- [Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAG](https://mlsanity.com/paper/2604.12047/) (2026)
- [TASER: Table Agents for Schema-guided Extraction and Recommendation](https://mlsanity.com/paper/2508.13404/) (2025)
- [Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System](https://mlsanity.com/paper/2603.12638/) (2026)
- [Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation](https://mlsanity.com/paper/2604.00003/) (2026)
- [HUGO-CS: A Hybrid-Labeled, Uncertainty-Aware, General-Purpose, Observational Dataset for Cold Spray](https://mlsanity.com/paper/2605.04257/) (2026)

## Abstract (arXiv)

Extracting information from academic PDF documents is crucial for numerous indexing, retrieval, and analysis use cases. Choosing the best tool to extract specific content elements is difficult because many, technically diverse tools are available, but recent performance benchmarks are rare. Moreover, such benchmarks typically cover only a few content elements like header metadata or bibliographic references and use smaller datasets from specific academic disciplines. We provide a large and diverse evaluation framework that supports more extraction tasks than most related datasets. Our framework builds upon DocBank, a multi-domain dataset of 1.5M annotated content elements extracted from 500K pages of research papers on arXiv. Using the new framework, we benchmark ten freely available tools in extracting document metadata, bibliographic references, tables, and other content elements from academic PDF documents. GROBID achieves the best metadata and reference extraction results, followed by CERMINE and Science Parse. For table extraction, Adobe Extract outperforms other tools, even though the performance is much lower than for other content elements. All tools struggle to extract lists, footers, and equations. We conclude that more research on improving and combining tools is necessary to achieve satisfactory extraction quality for most content elements. Evaluation datasets and frameworks like the one we present support this line of research. We make our data and code publicly available to contribute toward this goal.