# Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

arXiv: https://arxiv.org/abs/2605.27750 · HTML: https://mlsanity.com/paper/2605.27750/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 26 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 17 of 17)

- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [DeepSeek-OCR 2: Visual Causal Flow](https://mlsanity.com/paper/2601.20552/) (2026)
- [LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR](https://mlsanity.com/paper/2601.14251/) (2026)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR](https://mlsanity.com/paper/2601.03714/) (2026)
- [Optical Character Recognition of 19th Century Classical Commentaries: the Current State of Affairs](https://mlsanity.com/paper/2110.06817/) (2021)
- [Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study](https://mlsanity.com/paper/2412.20613/) (2024)
- [Important New Developments in Arabographic Optical Character Recognition (OCR)](https://mlsanity.com/paper/1703.09550/) (2017)
- [Ancient Script Image Recognition and Processing: A Review](https://mlsanity.com/paper/2506.19208/) (2025)
- [Structure-Aware Text Recognition for Ancient Greek Critical Editions](https://mlsanity.com/paper/2603.02803/) (2026)
- [GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts](https://mlsanity.com/paper/2604.12978/) (2026)
- [Advances and Limitations in Open Source Arabic-Script OCR: A Case Study](https://mlsanity.com/paper/2402.10943/) (2024)
- [Logios : An open source Greek Polytonic Optical Character Recognition system](https://mlsanity.com/paper/2506.21474/) (2025)
- [Geometric Risk Control for Vision-Language Model OCR](https://mlsanity.com/paper/2603.19790/) (2026)

## Built on by (most cited first, 5 of 5)

- [Using OCR Heads to Verbalize Image Semantics](https://mlsanity.com/paper/2609.18823/) (2026)
- [A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition](https://mlsanity.com/paper/2609.20064/) (2026)
- [When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context](https://mlsanity.com/paper/2609.34781/) (2026)
- [Which papyrus HTR is good enough? Character-error-rate tolerance of four papyrological tasks on Greek texts](https://mlsanity.com/paper/2609.37755/) (2026)
- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually unsupported text, suggesting reliance on language priors. Comparing open-weight VLMs with traditional OCR baselines on low-resource Ancient Greek critical editions, we show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions where traditional engines produce local recognition noise. To analyze visual evidence during decoding, we introduce controlled image perturbations and token-level grounding measures based on conditional versus image-free decoding distributions. Under character-level perturbations, VLMs diverge sharply from the perturbed ground truth while traditional OCR remains comparatively faithful; however, token-level analysis shows that prior reliance is model-specific: in an OCR-specialist model, fluent lexical errors are produced with little reliance on the image, whereas general-purpose VLMs remain conditioned on the visual input even when wrong. Decode-time interventions fail to reliably restore grounding, while post-OCR language-model correction improves several systems only by repairing text after generation. Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents and a broader set of models, showing that fluent output is not necessarily visually grounded and motivating interpretability-driven evaluation beyond aggregate accuracy.