# End-to-End Page-Level Assessment of Handwritten Text Recognition

arXiv: https://arxiv.org/abs/2301.05935 · HTML: https://mlsanity.com/paper/2301.05935/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 66 references that are in the map.
- First posted on arXiv 14 Jan 2023.

## Builds on (most cited first, 12 of 12)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition](https://mlsanity.com/paper/1604.08352/) (2016)
- [CloudScan - A configuration-free invoice analysis system using recurrent neural networks](https://mlsanity.com/paper/1708.07403/) (2017)
- [Full Page Handwriting Recognition via Image to Sequence Extraction](https://mlsanity.com/paper/2103.06450/) (2021)
- [ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction](https://mlsanity.com/paper/2106.10786/) (2021)
- [SPAN: a Simple Predict & Align Network for Handwritten Paragraph Recognition](https://mlsanity.com/paper/2102.08742/) (2021)
- [PageNet: Towards End-to-End Weakly Supervised Page-Level Handwritten Chinese Text Recognition](https://mlsanity.com/paper/2207.14807/) (2022)
- [Evaluation of HTR models without Ground Truth Material](https://mlsanity.com/paper/2201.06170/) (2022)
- [End-To-End Measure for Text Recognition](https://mlsanity.com/paper/1908.09584/) (2019)
- [Evaluation of a Region Proposal Architecture for Multi-task Document Layout Analysis](https://mlsanity.com/paper/2106.11797/) (2021)

## Built on by (most cited first, 8 of 8)

- [Handwritten Text Recognition: A Survey](https://mlsanity.com/paper/2502.08417/) (2025)
- [Class-Aware Mask-Guided Feature Refinement for Scene Text Recognition](https://mlsanity.com/paper/2402.13643/) (2024)
- [A Human-in-the-Loop Label Error Detection Framework Applied to Arabic-Script HTR Datasets](https://mlsanity.com/paper/2601.16713/) (2026)
- [Reading Order Independent Metrics for Information Extraction in Handwritten Documents](https://mlsanity.com/paper/2404.18664/) (2024)
- [Self-supervised Pre-training of Text Recognizers](https://mlsanity.com/paper/2405.00420/) (2024)
- [Spatial Context-based Self-Supervised Learning for Handwritten Text Recognition](https://mlsanity.com/paper/2404.11585/) (2024)
- [Align, Minimize and Diversify: A Source-Free Unsupervised Domain Adaptation Method for Handwritten Text Recognition](https://mlsanity.com/paper/2404.18260/) (2024)
- [DEEP: Docker-based Execution and Evaluation Platform](https://mlsanity.com/paper/2602.19583/) (2026)

## Abstract (arXiv)

The evaluation of Handwritten Text Recognition (HTR) systems has traditionally used metrics based on the edit distance between HTR and ground truth (GT) transcripts, at both the character and word levels. This is very adequate when the experimental protocol assumes that both GT and HTR text lines are the same, which allows edit distances to be independently computed to each given line. Driven by recent advances in pattern recognition, HTR systems increasingly face the end-to-end page-level transcription of a document, where the precision of locating the different text lines and their corresponding reading order (RO) play a key role. In such a case, the standard metrics do not take into account the inconsistencies that might appear. In this paper, the problem of evaluating HTR systems at the page level is introduced in detail. We analyse the convenience of using a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately. Different alternatives are proposed, analysed and empirically compared both through partially simulated and through real, full end-to-end experiments. Results support the validity of the proposed two-fold evaluation approach. An important conclusion is that such an evaluation can be adequately achieved by just two simple and well-known metrics: the Word Error Rate (WER), that takes transcription sequentiality into account, and the here re-formulated Bag of Words Word Error Rate (bWER), that ignores order. While the latter directly and very accurately assess intrinsic word recognition errors, the difference between both metrics gracefully correlates with the Normalised Spearman's Foot Rule Distance (NSFD), a metric which explicitly measures RO errors associated with layout analysis flaws.