# Aligning benchmark datasets for table structure recognition

arXiv: https://arxiv.org/abs/2303.00716 · HTML: https://mlsanity.com/paper/2303.00716/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context](https://mlsanity.com/paper/2005.00589/) (2020)
- [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://mlsanity.com/paper/2110.00061/) (2021)
- [TableBank: A Benchmark Dataset for Table Detection and Recognition](https://mlsanity.com/paper/1903.01949/) (2019)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents](https://mlsanity.com/paper/2004.12629/) (2020)
- [TableFormer: Table Structure Understanding with Transformers](https://mlsanity.com/paper/2203.01017/) (2022)
- [Complicated Table Structure Recognition](https://mlsanity.com/paper/1908.04729/) (2019)
- [Neural Collaborative Graph Machines for Table Structure Recognition](https://mlsanity.com/paper/2111.13359/) (2021)
- [GriTS: Grid table similarity metric for table structure recognition](https://mlsanity.com/paper/2203.12555/) (2022)
- [Current Status and Performance Analysis of Table Recognition in Document Images with Deep Neural Networks](https://mlsanity.com/paper/2104.14272/) (2021)
- [Guided Table Structure Recognition through Anchor Optimization](https://mlsanity.com/paper/2104.10538/) (2021)

## Built on by (most cited first, 15 of 15)

- [SEMv2: Table Separation Line Detection Based on Instance Segmentation](https://mlsanity.com/paper/2303.04384/) (2023)
- [SPRINT: Script-agnostic Structure Recognition in Tables](https://mlsanity.com/paper/2503.11932/) (2025)
- [RAPTOR: Refined Approach for Product Table Object Recognition](https://mlsanity.com/paper/2502.14918/) (2025)
- [LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition](https://mlsanity.com/paper/2607.13347/) (2026)
- [PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction](https://mlsanity.com/paper/2512.10888/) (2025)
- [Rethinking Detection Based Table Structure Recognition for Visually Rich Document Images](https://mlsanity.com/paper/2312.00699/) (2023)
- [TableCenterNet: A one-stage network for table structure recognition](https://mlsanity.com/paper/2504.17522/) (2025)
- [Spatial ModernBERT: Spatial-Aware Transformer for Table and Key-Value Extraction in Financial Documents at Scale](https://mlsanity.com/paper/2507.08865/) (2025)
- [Collage: Decomposable Rapid Prototyping for Information Extraction on Scientific PDFs](https://mlsanity.com/paper/2410.23478/) (2024)
- [CISOL: An Open and Extensible Dataset for Table Structure Recognition in the Construction Industry](https://mlsanity.com/paper/2501.15469/) (2025)
- [Hierarchical structure understanding in complex tables with VLLMs: a benchmark and experiments](https://mlsanity.com/paper/2511.08298/) (2025)
- [TableSeq: Unified Generation of Structure, Content, and Layout](https://mlsanity.com/paper/2604.16070/) (2026)
- [POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction](https://mlsanity.com/paper/2606.09788/) (2026)
- [ConRTF: Edge-Constrained Boundary Distribution Refinement for Realtime TransFormer Table Structure Recognition](https://mlsanity.com/paper/2607.00734/) (2026)
- [Tables Decoded: DELTA for Structure, TARQA for Understanding](https://mlsanity.com/paper/2609.17458/) (2026)

## Abstract (arXiv)

Benchmark datasets for table structure recognition (TSR) must be carefully processed to ensure they are annotated consistently. However, even if a dataset's annotations are self-consistent, there may be significant inconsistency across datasets, which can harm the performance of models trained and evaluated on them. In this work, we show that aligning these benchmarksunicodex2014removing both errors and inconsistency between themunicodex2014improves model performance significantly. We demonstrate this through a data-centric approach where we adopt one model architecture, the Table Transformer (TATR), that we hold fixed throughout. Baseline exact match accuracy for TATR evaluated on the ICDAR-2013 benchmark is 65% when trained on PubTables-1M, 42% when trained on FinTabNet, and 69% combined. After reducing annotation mistakes and inter-dataset inconsistency, performance of TATR evaluated on ICDAR-2013 increases substantially to 75% when trained on PubTables-1M, 65% when trained on FinTabNet, and 81% combined. We show through ablations over the modification steps that canonicalization of the table annotations has a significantly positive effect on performance, while other choices balance necessary trade-offs that arise when deciding a benchmark dataset's final composition. Overall we believe our work has significant implications for benchmark design for TSR and potentially other tasks as well. Dataset processing and training code will be released at https://github.com/microsoft/table-transformer.