# Benchmarking Large Language Models for Handwritten Text Recognition

arXiv: https://arxiv.org/abs/2503.15195 · HTML: https://mlsanity.com/paper/2503.15195/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 22 of 22)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems](https://mlsanity.com/paper/2310.12397/) (2023)
- [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [OrigamiNet: Weakly-Supervised, Segmentation-Free, One-Step, Full Page Text Recognition by learning to unfold](https://mlsanity.com/paper/2006.07491/) (2020)
- [End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network](https://mlsanity.com/paper/2012.03868/) (2020)
- [Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention](https://mlsanity.com/paper/1604.03286/) (2016)
- [Full Page Handwriting Recognition via Image to Sequence Extraction](https://mlsanity.com/paper/2103.06450/) (2021)
- [HTR-VT: Handwritten Text Recognition with Vision Transformer](https://mlsanity.com/paper/2409.08573/) (2024)
- [The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognition](https://mlsanity.com/paper/2208.07682/) (2022)
- [Handwriting Recognition of Historical Documents with few labeled data](https://mlsanity.com/paper/1811.07768/) (2018)
- [Transformer-based HTR for Historical Documents](https://mlsanity.com/paper/2203.11008/) (2022)
- [Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records](https://mlsanity.com/paper/2501.11623/) (2025)
- [Archives and AI: An Overview of Current Debates and Future Perspectives](https://mlsanity.com/paper/2105.01117/) (2021)
- [Evaluation of HTR models without Ground Truth Material](https://mlsanity.com/paper/2201.06170/) (2022)
- [CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models](https://mlsanity.com/paper/2408.17428/) (2024)
- [Handwriting Recognition in Historical Documents with Multimodal LLM](https://mlsanity.com/paper/2410.24034/) (2024)

## Built on by (most cited first, 11 of 11)

- [Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities](https://mlsanity.com/paper/2510.06743/) (2025)
- [Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877-1918)](https://mlsanity.com/paper/2512.19675/) (2025)
- [METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition](https://mlsanity.com/paper/2605.26712/) (2026)
- [An HTR-LLM Workflow for High-Accuracy Transcription and Analysis of Abbreviated Latin Court Hand](https://mlsanity.com/paper/2507.04132/) (2025)
- [Improving MLLM Historical Record Extraction with Test-Time Image](https://mlsanity.com/paper/2509.09722/) (2025)
- [JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding](https://mlsanity.com/paper/2603.27942/) (2026)
- [Handwriting Extraction and Analysis of Signature Lists in Swiss Popular Initiatives](https://mlsanity.com/paper/2606.05018/) (2026)
- [When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study](https://mlsanity.com/paper/2608.22366/) (2026)
- [OCR-Based Field Extraction for Archaeological Pottery Metadata: The CENTURIA Dataset](https://mlsanity.com/paper/2608.30616/) (2026)
- [ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts](https://mlsanity.com/paper/2609.12705/) (2026)
- [Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education](https://mlsanity.com/paper/2609.14284/) (2026)

## Abstract (arXiv)

Traditional machine learning models for Handwritten Text Recognition (HTR) rely on supervised training, requiring extensive manual annotations, and often produce errors due to the separation between layout and text processing. In contrast, Multimodal Large Language Models (MLLMs) offer a general approach to recognizing diverse handwriting styles without the need for model-specific training. The study benchmarks various proprietary and open-source LLMs against Transkribus models, evaluating their performance on both modern and historical datasets written in English, French, German, and Italian. In addition, emphasis is placed on testing the models' ability to autonomously correct previously generated outputs. Findings indicate that proprietary models, especially Claude 3.5 Sonnet, outperform open-source alternatives in zero-shot settings. MLLMs achieve excellent results in recognizing modern handwriting and exhibit a preference for the English language due to their pre-training dataset composition. Comparisons with Transkribus show no consistent advantage for either approach. Moreover, LLMs demonstrate limited ability to autonomously correct errors in zero-shot transcriptions.