# Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents

arXiv: https://arxiv.org/abs/2504.00414 · HTML: https://mlsanity.com/paper/2504.00414/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 99 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 29 of 29)

- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis](https://mlsanity.com/paper/2103.15348/) (2021)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study](https://mlsanity.com/paper/2401.10825/) (2024)
- [Transformer-Based Approach for Joint Handwriting and Named Entity Recognition in Historical documents](https://mlsanity.com/paper/2112.04189/) (2021)
- [OCR4all -- An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings](https://mlsanity.com/paper/1909.04032/) (2019)
- [OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches](https://mlsanity.com/paper/2502.01205/) (2025)
- [Transformer-based HTR for Historical Documents](https://mlsanity.com/paper/2203.11008/) (2022)
- [Yes but.. Can ChatGPT Identify Entities in Historical Documents?](https://mlsanity.com/paper/2303.17322/) (2023)
- [Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records](https://mlsanity.com/paper/2501.11623/) (2025)
- [An Unsupervised method for OCR Post-Correction and Spelling Normalisation for Finnish](https://mlsanity.com/paper/2011.03502/) (2020)
- [DANIEL: A fast Document Attention Network for Information Extraction and Labelling of handwritten documents](https://mlsanity.com/paper/2407.09103/) (2024)
- [OCR of historical printings with an application to building diachronic corpora: A case study using the RIDGES herbal corpus](https://mlsanity.com/paper/1608.02153/) (2016)
- [Mixed Model OCR Training on Historical Latin Script for Out-of-the-Box Recognition and Finetuning](https://mlsanity.com/paper/2106.07881/) (2021)
- [CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models](https://mlsanity.com/paper/2408.17428/) (2024)
- [State of the Art Optical Character Recognition of 19th Century Fraktur Scripts using Open Source Engines](https://mlsanity.com/paper/1810.03436/) (2018)
- [Processing the structure of documents: Logical Layout Analysis of historical newspapers in French](https://mlsanity.com/paper/2202.08125/) (2022)
- [End-to-end information extraction in handwritten documents: Understanding Paris marriage records from 1880 to 1940](https://mlsanity.com/paper/2404.19329/) (2024)
- [Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts](https://mlsanity.com/paper/2412.16119/) (2024)
- [NER4all or Context is All You Need: Using LLMs for low-effort, high-performance NER on historical texts. A humanities informed approach](https://mlsanity.com/paper/2502.04351/) (2025)
- [A Benchmark of Nested Named Entity Recognition Approaches in Historical Structured Documents](https://mlsanity.com/paper/2302.10204/) (2023)
- [Improving OCR Quality in 19th Century Historical Documents Using a Combined Machine Learning Based Approach](https://mlsanity.com/paper/2401.07787/) (2024)
- [You Actually Look Twice At it (YALTAi): using an object detection approach instead of region segmentation within the Kraken engine](https://mlsanity.com/paper/2207.11230/) (2022)
- [Enhancing OCR Performance through Post-OCR Models: Adopting Glyph Embedding for Improved Correction](https://mlsanity.com/paper/2308.15262/) (2023)
- [Handwriting Recognition in Historical Documents with Multimodal LLM](https://mlsanity.com/paper/2410.24034/) (2024)
- [HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis](https://mlsanity.com/paper/2412.18981/) (2024)
- [Leveraging large language models for structured information extraction from pathology reports](https://mlsanity.com/paper/2502.12183/) (2025)

## Built on by (most cited first, 12 of 12)

- [Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877-1918)](https://mlsanity.com/paper/2512.19675/) (2025)
- [OmniOCR: Generalist OCR for Ethnic Minority Languages](https://mlsanity.com/paper/2602.21042/) (2026)
- [GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts](https://mlsanity.com/paper/2604.12978/) (2026)
- [A Language Model from 1913: Pretraining on Historical Text](https://mlsanity.com/paper/2606.02991/) (2026)
- [TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction](https://mlsanity.com/paper/2604.22880/) (2026)
- [An HTR-LLM Workflow for High-Accuracy Transcription and Analysis of Abbreviated Latin Court Hand](https://mlsanity.com/paper/2507.04132/) (2025)
- [Leveraging LLMs for Context-Aware Implicit Textual and Multimodal Hate Speech Detection](https://mlsanity.com/paper/2510.15685/) (2025)
- [The Devil is in the Details -- From OCR for Old Church Slavonic to Purely Visual Stemma Reconstruction](https://mlsanity.com/paper/2604.11724/) (2026)
- [DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselines](https://mlsanity.com/paper/2604.14314/) (2026)
- [MUDIDI: A Two-Stage Framework for Multilingual Dictionary Digitization with Language Models](https://mlsanity.com/paper/2606.09435/) (2026)
- [When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study](https://mlsanity.com/paper/2608.22366/) (2026)
- [Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction](https://mlsanity.com/paper/2609.21595/) (2026)

## Abstract (arXiv)

We explore how multimodal Large Language Models (mLLMs) can help researchers transcribe historical documents, extract relevant historical information, and construct datasets from historical sources. Specifically, we investigate the capabilities of mLLMs in performing (1) Optical Character Recognition (OCR), (2) OCR Post-Correction, and (3) Named Entity Recognition (NER) tasks on a set of city directories published in German between 1754 and 1870. First, we benchmark the off-the-shelf transcription accuracy of both mLLMs and conventional OCR models. We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs. Second, we are the first to introduce multimodal post-correction of OCR output using mLLMs. We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning. Third, we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats. Our findings provide early evidence for the long-term potential of mLLMs to introduce a paradigm shift in the approaches to historical data collection and document transcription.