# Reading Order Independent Metrics for Information Extraction in Handwritten Documents

arXiv: https://arxiv.org/abs/2404.18664 · HTML: https://mlsanity.com/paper/2404.18664/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 14 of 14)

- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://mlsanity.com/paper/2108.04539/) (2021)
- [LayoutReader: Pre-training of Text and Layout for Reading Order Detection](https://mlsanity.com/paper/2108.11591/) (2021)
- [Spatial Dependency Parsing for Semi-Structured Document Information Extraction](https://mlsanity.com/paper/2005.00642/) (2020)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction](https://mlsanity.com/paper/2310.11016/) (2023)
- [Key-value information extraction from full handwritten pages](https://mlsanity.com/paper/2304.13530/) (2023)
- [End-to-End Page-Level Assessment of Handwritten Text Recognition](https://mlsanity.com/paper/2301.05935/) (2023)
- [Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model](https://mlsanity.com/paper/1803.06252/) (2018)
- [Large Scale Genealogical Information Extraction From Handwritten Quebec Parish Records](https://mlsanity.com/paper/2304.14044/) (2023)

## Built on by (most cited first, 2 of 2)

- [Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Review](https://mlsanity.com/paper/2408.06345/) (2024)
- [Taming LLMs with Negative Samples: A Reference-Free Framework to Evaluate Presentation Content with Actionable Feedback](https://mlsanity.com/paper/2505.18240/) (2025)

## Abstract (arXiv)

Information Extraction processes in handwritten documents tend to rely on obtaining an automatic transcription and performing Named Entity Recognition (NER) over such transcription. For this reason, in publicly available datasets, the performance of the systems is usually evaluated with metrics particular to each dataset. Moreover, most of the metrics employed are sensitive to reading order errors. Therefore, they do not reflect the expected final application of the system and introduce biases in more complex documents. In this paper, we propose and publicly release a set of reading order independent metrics tailored to Information Extraction evaluation in handwritten documents. In our experimentation, we perform an in-depth analysis of the behavior of the metrics to recommend what we consider to be the minimal set of metrics to evaluate a task correctly.