# HATFormer: Historic Handwritten Arabic Text Recognition with Transformers

arXiv: https://arxiv.org/abs/2410.02179 · HTML: https://mlsanity.com/paper/2410.02179/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [Decoupled Attention Network for Text Recognition](https://mlsanity.com/paper/1912.10205/) (2019)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition](https://mlsanity.com/paper/1903.07377/) (2019)
- [DTrOCR: Decoder-only Transformer for Optical Character Recognition](https://mlsanity.com/paper/2308.15996/) (2023)
- [Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition](https://mlsanity.com/paper/2406.09630/) (2024)
- [A Transformer-based Approach for Arabic Offline Handwritten Text Recognition](https://mlsanity.com/paper/2307.15045/) (2023)
- [Representing Online Handwriting for Recognition in Large Vision-Language Models](https://mlsanity.com/paper/2402.15307/) (2024)

## Built on by (most cited first, 9 of 9)

- [A Human-in-the-Loop Label Error Detection Framework Applied to Arabic-Script HTR Datasets](https://mlsanity.com/paper/2601.16713/) (2026)
- [Cross-Lingual Learning within Arabic Script for Low-Resource HTR](https://mlsanity.com/paper/2605.02089/) (2026)
- [RefLAM: A Reference-Grounded Line Annotation Pipeline for Historical Arabic Manuscripts](https://mlsanity.com/paper/2608.25140/) (2026)
- [Different Strokes for Different Folks: Writer Identification for Historical Arabic Manuscripts](https://mlsanity.com/paper/2604.22515/) (2026)
- [Performance Gap Analysis between Latin and Arabic Scripts HTR](https://mlsanity.com/paper/2606.18884/) (2026)
- [Does a Modern-Handwriting Warm-Up Help Historical Arabic OCR? A Reproducible, Compute-Matched Evaluation on Muharaf and KHATT](https://mlsanity.com/paper/2608.22316/) (2026)
- [When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study](https://mlsanity.com/paper/2608.22366/) (2026)
- [AraMS-28k: The Largest Publicly Released Line-Level Dataset of Historical Arabic Manuscripts with Margin and Insertion-Anchor Annotations](https://mlsanity.com/paper/2608.26921/) (2026)
- [HPMD: A Historical Persian Manuscript Dataset for Word Spotting with Line-Level Annotation](https://mlsanity.com/paper/2609.34490/) (2026)

## Abstract (arXiv)

Arabic handwritten text recognition (HTR) is challenging, especially for historical texts, due to diverse writing styles and the intrinsic features of Arabic script. Additionally, Arabic handwriting datasets are smaller compared to English ones, making it difficult to train generalizable Arabic HTR models. To address these challenges, we propose HATFormer, a transformer-based encoder-decoder architecture that builds on a state-of-the-art English HTR model. By leveraging the transformer's attention mechanism, HATFormer captures spatial contextual information to address the intrinsic challenges of Arabic script through differentiating cursive characters, decomposing visual representations, and identifying diacritics. Our customization to historical handwritten Arabic includes an image processor for effective ViT information preprocessing, a text tokenizer for compact Arabic text representation, and a training pipeline that accounts for a limited amount of historic Arabic handwriting data. HATFormer achieves a character error rate (CER) of 8.6% on the largest public historical handwritten Arabic dataset, with a 51% improvement over the best baseline in the literature. HATFormer also attains a comparable CER of 4.2% on the largest private non-historical dataset. Our work demonstrates the feasibility of adapting an English HTR method to a low-resource language with complex, language-specific challenges, contributing to advancements in document digitization, information retrieval, and cultural preservation.