# Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities

arXiv: https://arxiv.org/abs/2510.06743 · HTML: https://mlsanity.com/paper/2510.06743/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 6 of 6)

- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Benchmarking Large Language Models for Handwritten Text Recognition](https://mlsanity.com/paper/2503.15195/) (2025)
- [Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records](https://mlsanity.com/paper/2501.11623/) (2025)
- [Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts](https://mlsanity.com/paper/2412.16119/) (2024)
- [Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual Complexity](https://mlsanity.com/paper/2503.23667/) (2025)
- [Unlocking the Archives: Using Large Language Models to Transcribe Handwritten Historical Documents](https://mlsanity.com/paper/2411.03340/) (2024)

## Built on by (most cited first, 5 of 5)

- [Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications](https://mlsanity.com/paper/2602.16201/) (2026)
- [Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877-1918)](https://mlsanity.com/paper/2512.19675/) (2025)
- [Quid est VERITAS? A Modular Framework for Archival Document Analysis](https://mlsanity.com/paper/2603.28108/) (2026)
- [The Devil is in the Details -- From OCR for Old Church Slavonic to Purely Visual Stemma Reconstruction](https://mlsanity.com/paper/2604.11724/) (2026)
- [When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study](https://mlsanity.com/paper/2608.22366/) (2026)

## Abstract (arXiv)

Digital humanities scholars increasingly use Large Language Models for historical document digitization, yet lack appropriate evaluation frameworks for LLM-based OCR. Traditional metrics fail to capture temporal biases and period-specific errors crucial for historical corpus creation. We present an evaluation methodology for LLM-based historical OCR, addressing contamination risks and systematic biases in diplomatic transcription. Using 18th-century Russian Civil font texts, we introduce novel metrics including Historical Character Preservation Rate (HCPR) and Archaic Insertion Rate (AIR), alongside protocols for contamination control and stability testing. We evaluate 12 multimodal LLMs, finding that Gemini and Qwen models outperform traditional OCR while exhibiting over-historicization: inserting archaic characters from incorrect historical periods. Post-OCR correction degrades rather than improves performance. Our methodology provides digital humanities practitioners with guidelines for model selection and quality assessment in historical corpus digitization.