# From Historical Tabular Image to Knowledge Graphs: A Provenance-Aware Modular Pipeline

arXiv: https://arxiv.org/abs/2605.08222 · HTML: https://mlsanity.com/paper/2605.08222/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 6 May 2026.

## Builds on (most cited first, 14 of 14)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities](https://mlsanity.com/paper/2305.13168/) (2023)
- [Complicated Table Structure Recognition](https://mlsanity.com/paper/1908.04729/) (2019)
- [Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning](https://mlsanity.com/paper/2304.02711/) (2023)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [LORE: Logical Location Regression Network for Table Structure Recognition](https://mlsanity.com/paper/2303.03730/) (2023)
- [Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner](https://mlsanity.com/paper/2412.20662/) (2024)
- [Uncertainty Management in the Construction of Knowledge Graphs: a Survey](https://mlsanity.com/paper/2405.16929/) (2024)
- [DocSpiral: A Platform for Integrated Assistive Document Annotation through Human-in-the-Spiral](https://mlsanity.com/paper/2505.03214/) (2025)
- [Ontologies in CLARIAH: Towards Interoperability in History, Language and Media](https://mlsanity.com/paper/2004.02845/) (2020)
- [Hybrid Intelligence for Digital Humanities](https://mlsanity.com/paper/2406.15374/) (2024)
- [Enhancing Data Integrity through Provenance Tracking in Semantic Web Frameworks](https://mlsanity.com/paper/2501.09029/) (2025)

## Abstract (arXiv)

Handwritten archival tables contain rich historical information, yet transforming them into structured representations, such as Knowledge Graphs, requires integrating table structure recognition, handwriting recognition, and semantic interpretation - a complex multimodal process. End-to-end AI implementations can obscure these steps, resulting in opaque algorithmic operations that hinder human oversight, critical assessment, and trust. To address this, we present a modular, provenance-aware pipeline to convert handwritten tabular images into KGs supporting human-AI collaboration. The pipeline decomposes the workflow into three stages - table reconstruction, information extraction, and KG construction - while exposing intermediate representations for inspection, evaluation, and correction. A key contribution of our approach is the systematic integration of data provenance at every stage, ensuring that all extracted entities and literals remain traceable to their visual and textual origins. The proposed pipeline is demonstrated through a number of experiments on real-world archival material concerning military careers. The results across three different table reconstruction variants highlight the importance of modularisation. By coupling modularity with data provenance, our work advances transparent and collaboratively controllable image-to-KG pipelines for complex historical data.