# SCORE: A Semantic Evaluation Framework for Generative Document Parsing

arXiv: https://arxiv.org/abs/2509.19345 · HTML: https://mlsanity.com/paper/2509.19345/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 14 of 14)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://mlsanity.com/paper/2110.00061/) (2021)
- [CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy](https://mlsanity.com/paper/2412.02210/) (2024)
- [Improving Table Structure Recognition with Visual-Alignment Sequential Coordinate Modeling](https://mlsanity.com/paper/2303.06949/) (2023)
- [DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems](https://mlsanity.com/paper/2407.10701/) (2024)
- [A large-scale dataset for end-to-end table recognition in the wild](https://mlsanity.com/paper/2303.14884/) (2023)
- [ICDAR 2021 Competition on Scientific Literature Parsing](https://mlsanity.com/paper/2106.14616/) (2021)
- [Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding](https://mlsanity.com/paper/2409.19672/) (2024)

## Built on by (most cited first, 5 of 5)

- [DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM](https://mlsanity.com/paper/2512.10619/) (2025)
- [Beyond String Matching: Semantic Evaluation of PDF Table Extraction](https://mlsanity.com/paper/2603.18652/) (2026)
- [Chronos: The AI Co-Historian](https://mlsanity.com/paper/2604.03553/) (2026)
- [Improving Layout Representation Learning Across Inconsistently Annotated Datasets via Agentic Harmonization](https://mlsanity.com/paper/2604.11042/) (2026)
- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

Multi-modal generative document parsing systems challenge traditional evaluation: unlike deterministic OCR or layout models, they often produce semantically correct yet structurally divergent outputs. Conventional metrics-CER, WER, IoU, or TEDS-misclassify such diversity as error, penalizing valid interpretations and obscuring system behavior. We introduce SCORE (Structural and COntent Robust Evaluation), an interpretation-agnostic framework that integrates (i) adjusted edit distance for robust content fidelity, (ii) token-level diagnostics to distinguish hallucinations from omissions, (iii) table evaluation with spatial tolerance and semantic alignment, and (iv) hierarchy-aware consistency checks. Together, these dimensions enable evaluation that embraces representational diversity while enforcing semantic rigor. Across 1,114 pages spanning a holistic benchmark and a field dataset, SCORE consistently revealed cross-dataset performance patterns missed by standard metrics. In 2-5% of pages with ambiguous table structures, traditional metrics penalized systems by 12-25% on average, leading to distorted rankings. SCORE corrected these cases, recovering equivalence between alternative but valid interpretations. Moreover, by normalizing generative outputs into a format-agnostic representation, SCORE reproduces traditional scores (e.g., table F1 up to 0.93) without requiring object-detection pipelines, demonstrating that generative parsing alone suffices for comprehensive evaluation. By exposing how interpretive diversity impacts evaluation outcomes and providing multi-dimensional, interpretable diagnostics, SCORE establishes foundational principles for semantically grounded, fair, and practical benchmarking of modern document parsing systems.