# The Character Error Vector: Decomposable errors for page-level OCR evaluation

arXiv: https://arxiv.org/abs/2604.06160 · HTML: https://mlsanity.com/paper/2604.06160/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 20 of 20)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [PaddleOCR 3.0 Technical Report](https://mlsanity.com/paper/2507.05595/) (2025)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [LayoutReader: Pre-training of Text and Layout for Reading Order Detection](https://mlsanity.com/paper/2108.11591/) (2021)
- [dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model](https://mlsanity.com/paper/2512.02498/) (2025)
- [PP-DocLayout: A Unified Document Layout Detection Model to Accelerate Large-Scale Data Construction](https://mlsanity.com/paper/2503.17213/) (2025)
- [GLM-OCR Technical Report](https://mlsanity.com/paper/2603.10910/) (2026)
- [PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks](https://mlsanity.com/paper/2603.24373/) (2026)
- [Advanced Layout Analysis Models for Docling](https://mlsanity.com/paper/2509.11720/) (2025)
- [DISGO: Automatic End-to-End Evaluation for Scene Text OCR](https://mlsanity.com/paper/2308.13173/) (2023)
- [Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models](https://mlsanity.com/paper/2502.14901/) (2025)
- [LED Benchmark: Diagnosing Structural Layout Errors for Document Layout Analysis](https://mlsanity.com/paper/2507.23295/) (2025)
- [Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCR](https://mlsanity.com/paper/2410.03536/) (2024)
- [Tracking Amendments to Legislation and Other Political Texts with a Novel Minimum-Edit-Distance Algorithm: DocuToads](https://mlsanity.com/paper/1608.06459/) (2016)
- [The COTe score: A decomposable framework for evaluating Document Layout Analysis models](https://mlsanity.com/paper/2603.12718/) (2026)

## Built on by (most cited first, 2 of 2)

- [OCR-Based Field Extraction for Archaeological Pottery Metadata: The CENTURIA Dataset](https://mlsanity.com/paper/2608.30616/) (2026)
- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

The Character Error Rate (CER) is a key metric for evaluating the quality of Optical Character Recognition (OCR). However, this metric assumes that text has been perfectly parsed, which is often not the case. Under page-parsing errors, CER becomes undefined, limiting its use as a metric and making evaluating page-level OCR challenging, particularly when using data that do not share a labelling schema. We introduce the Character Error Vector (CEV), a bag-of-characters evaluator for OCR. The CEV can be decomposed into parsing and OCR, and interaction error components. This decomposability allows practitioners to focus on the part of the Document Understanding pipeline that will have the greatest impact on overall text extraction quality. The CEV can be implemented using a variety of methods, of which we demonstrate SpACER (Spatially Aware Character Error Rate) and a Character distribution method using the Jensen-Shannon Distance. We validate the CEV's performance against other metrics: first, the relationship with CER; then, parse quality; and finally, as a direct measure of page-level OCR quality. The validation process shows that the CEV is a valuable bridge between parsing metrics and local metrics like CER. We analyse a dataset of archival newspapers made of degraded images with complex layouts and find that state-of-the-art end-to-end models are outperformed by more traditional pipeline approaches. Whilst the CEV requires character-level positioning for optimal triage, thresholding on easily available values can predict the main error source with an F1 of 0.91. We provide the CEV as part of a Python library to support Document understanding research.