# Transformer-Based Approach for Joint Handwriting and Named Entity Recognition in Historical documents arXiv: https://arxiv.org/abs/2112.04189 ยท HTML: https://mlsanity.com/paper/2112.04189/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 27 references that are in the map. - First posted on arXiv 8 Dec 2021. ## Builds on (most cited first, 15 of 15) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019) - [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016) - [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020) - [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://mlsanity.com/paper/1910.04396/) (2019) - [Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition](https://mlsanity.com/paper/1903.07377/) (2019) - [OrigamiNet: Weakly-Supervised, Segmentation-Free, One-Step, Full Page Text Recognition by learning to unfold](https://mlsanity.com/paper/2006.07491/) (2020) - [End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network](https://mlsanity.com/paper/2012.03868/) (2020) - [Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention](https://mlsanity.com/paper/1604.03286/) (2016) - [DE-GAN: A Conditional Generative Adversarial Network for Document Enhancement](https://mlsanity.com/paper/2010.08764/) (2020) - [docExtractor: An off-the-shelf historical document element extraction](https://mlsanity.com/paper/2012.08191/) (2020) - [A Neural Model for Text Localization, Transcription and Named Entity Recognition in Full Pages](https://mlsanity.com/paper/1912.10016/) (2019) - [A Few-shot Learning Approach for Historical Ciphered Manuscript Recognition](https://mlsanity.com/paper/2009.12577/) (2020) - [Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model](https://mlsanity.com/paper/1803.06252/) (2018) ## Built on by (most cited first, 17 of 17) - [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022) - [DocEnTr: An End-to-End Document Image Enhancement Transformer](https://mlsanity.com/paper/2201.10252/) (2022) - [Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents](https://mlsanity.com/paper/2504.00414/) (2025) - [Faster DAN: Multi-target Queries with Document Positional Encoding for End-to-end Handwritten Document Recognition](https://mlsanity.com/paper/2301.10593/) (2023) - [Key-value information extraction from full handwritten pages](https://mlsanity.com/paper/2304.13530/) (2023) - [DANIEL: A fast Document Attention Network for Information Extraction and Labelling of handwritten documents](https://mlsanity.com/paper/2407.09103/) (2024) - [MSdocTr-Lite: A Lite Transformer for Full Page Multi-script Handwriting Recognition](https://mlsanity.com/paper/2303.13931/) (2023) - [FontTransformer: Few-shot High-resolution Chinese Glyph Image Synthesis via Stacked Transformers](https://mlsanity.com/paper/2210.06301/) (2022) - [Large Scale Genealogical Information Extraction From Handwritten Quebec Parish Records](https://mlsanity.com/paper/2304.14044/) (2023) - [FATURA: A Multi-Layout Invoice Image Dataset for Document Analysis and Understanding](https://mlsanity.com/paper/2311.11856/) (2023) - [End-to-end information extraction in handwritten documents: Understanding Paris marriage records from 1880 to 1940](https://mlsanity.com/paper/2404.19329/) (2024) - [ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical Handwritten Documents](https://mlsanity.com/paper/2303.03127/) (2023) - [A Review of Hybrid and Ensemble in Deep Learning for Natural Language Processing](https://mlsanity.com/paper/2312.05589/) (2023) - [DocTTT: Test-Time Training for Handwritten Document Recognition Using Meta-Auxiliary Learning](https://mlsanity.com/paper/2501.12898/) (2025) - [Meta-DAN: towards an efficient prediction strategy for page-level handwritten text recognition](https://mlsanity.com/paper/2504.03349/) (2025) - [MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning](https://mlsanity.com/paper/2505.20513/) (2025) - [SIMARA: a database for key-value information extraction from full pages](https://mlsanity.com/paper/2304.13606/) (2023) ## Abstract (arXiv) The extraction of relevant information carried out by named entities in handwriting documents is still a challenging task. Unlike traditional information extraction approaches that usually face text transcription and named entity recognition as separate subsequent tasks, we propose in this paper an end-to-end transformer-based approach to jointly perform these two tasks. The proposed approach operates at the paragraph level, which brings two main benefits. First, it allows the model to avoid unrecoverable early errors due to line segmentation. Second, it allows the model to exploit larger bi-dimensional context information to identify the semantic categories, reaching a higher final prediction accuracy. We also explore different training scenarios to show their effect on the performance and we demonstrate that a two-stage learning strategy can make the model reach a higher final prediction accuracy. As far as we know, this work presents the first approach that adopts the transformer networks for named entity recognition in handwritten documents. We achieve the new state-of-the-art performance in the ICDAR 2017 Information Extraction competition using the Esposalles database, for the complete task, even though the proposed technique does not use any dictionaries, language modeling, or post-processing.