# The COTe score: A decomposable framework for evaluating Document Layout Analysis models

arXiv: https://arxiv.org/abs/2603.12718 · HTML: https://mlsanity.com/paper/2603.12718/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Builds on (most cited first, 22 of 22)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [M6Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysis](https://mlsanity.com/paper/2305.08719/) (2023)
- [PP-DocLayout: A Unified Document Layout Detection Model to Accelerate Large-Scale Data Construction](https://mlsanity.com/paper/2503.17213/) (2025)
- [PP-PicoDet: A Better Real-Time Object Detector on Mobile Devices](https://mlsanity.com/paper/2111.00902/) (2021)
- [Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding](https://mlsanity.com/paper/2206.03484/) (2022)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Automatic universal taxonomies for multi-domain semantic segmentation](https://mlsanity.com/paper/2207.08445/) (2022)
- [ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents](https://mlsanity.com/paper/2305.14962/) (2023)
- [Advanced Layout Analysis Models for Docling](https://mlsanity.com/paper/2509.11720/) (2025)
- [General Information Metrics for Improving AI Model Training Efficiency](https://mlsanity.com/paper/2501.02004/) (2025)
- [ICDAR 2025 Competition on FEw-Shot Text line segmentation of ancient handwritten documents (FEST)](https://mlsanity.com/paper/2509.12965/) (2025)
- [Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models](https://mlsanity.com/paper/2502.14901/) (2025)
- [LED Benchmark: Diagnosing Structural Layout Errors for Document Layout Analysis](https://mlsanity.com/paper/2507.23295/) (2025)
- [Label Unification for Cross-Dataset Generalization in Cybersecurity NER](https://mlsanity.com/paper/2507.13870/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Character Error Vector: Decomposable errors for page-level OCR evaluation](https://mlsanity.com/paper/2604.06160/) (2026)

## Abstract (arXiv)

Document Layout Analysis (DLA) is the process by which a page is parsed into meaningful elements, often using machine learning models. Typically, the quality of a model is judged using general machine vision metrics such as IoU, F1 or mAP. However, these metrics are designed for images that are 2D projections of 3D space, not for the natively 2D imagery of printed media. This discrepancy can result in misleading or uninformative interpretation of model performance. To encourage more robust, comparable, and nuanced DLA, we introduce: The Structural Semantic Unit (SSU), a relational labelling approach that shifts the focus from the physical to the semantic structure of the content; and the Coverage, Overlap, Trespass, and Excess (COTe) score, a decomposable metric for measuring page parsing quality. We demonstrate the value of these methods through case studies and by evaluating 5 common DLA models on 3 DLA datasets. We show that the COTe score is more informative than traditional metrics and reveals distinct failure modes across models, such as breaching semantic boundaries or repeatedly parsing the same region. We find that, under granularity differences between model and ground truth, the COTe score is substantially more robust than the F1. Even in the worst case, comparing character-level predictions against paragraph-level ground truth with otherwise perfect parsing, COTe returns 0.68 where F1 returns 0. Notably, we find that, on real datasets, the COTe's granularity robustness largely holds even without explicit SSU labelling, reducing the barrier to entry. Finally, we release an SSU labelled dataset and a Python library for applying COTe in DLA projects.