# When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

arXiv: https://arxiv.org/abs/2607.24077 · HTML: https://mlsanity.com/paper/2607.24077/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 17 of 17)

- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [DeepSeek-OCR: Contexts Optical Compression](https://mlsanity.com/paper/2510.18234/) (2025)
- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing](https://mlsanity.com/paper/2601.21957/) (2026)
- [Qianfan-OCR: A Unified End-to-End Model for Document Intelligence](https://mlsanity.com/paper/2603.13398/) (2026)
- [Calamari - A High-Performance Tensorflow-based Deep Learning Package for Optical Character Recognition](https://mlsanity.com/paper/1807.02004/) (2018)
- [Multimodal OCR: Parse Anything from Documents](https://mlsanity.com/paper/2603.13032/) (2026)
- [How to Choose Pretrained Handwriting Recognition Models for Single Writer Fine-Tuning](https://mlsanity.com/paper/2305.02593/) (2023)
- [OCR4all -- An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings](https://mlsanity.com/paper/1909.04032/) (2019)
- [CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition](https://mlsanity.com/paper/2509.19768/) (2025)
- [Improving OCR using internal document redundancy](https://mlsanity.com/paper/2508.14557/) (2025)
- [Few-shot Writer Adaptation via Multimodal In-Context Learning](https://mlsanity.com/paper/2603.29450/) (2026)
- [Democratizing the medieval English legal tradition](https://mlsanity.com/paper/2605.00977/) (2026)
- [METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition](https://mlsanity.com/paper/2605.26712/) (2026)

## Built on by (most cited first, 1 of 1)

- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

Optical Character Recognition (OCR) is a key component in the digitization of historical archives. Recently, Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems, achieving state-of-the-art performance on standard benchmarks. However, their suitability for archival transcription remains insufficiently understood. In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset, a challenging collection of Uruguayan dictatorship-era documents derived from microfilm scans. While VLMs consistently outperform traditional methods in terms of Character Error Rate (CER) and Word Error Rate (WER), we show that these improvements hide a more complex picture. Through a detailed qualitative analysis, we uncover systematic failure modes that are invisible to standard metrics, including orthographic normalization, spurious content generation, and semantic substitutions that preserve fluency while altering meaning. Errors affecting named entities are particularly critical, as they can introduce substantial semantic distortions with minimal impact on CER and WER. These findings reveal a critical gap between quantitative OCR performance and transcription fidelity in real-world archival settings, and highlight the need for evaluation frameworks that go beyond character-level accuracy to capture the semantic reliability of generated transcriptions.