mlsanityData to 14 Oct 2026

When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context

arXiv 2609.34781 ↗First posted 28 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Document understanding and multimodal RAG

Abstract (arXiv)

Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45% to 58.51%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 23 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

21 of 23 references.

References · 21

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  3. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  4. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  6. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 116
  7. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Networkcited by 97
  8. 2023Revisiting Scene Text Recognition: A Data Perspectivecited by 45
  9. 2021Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Featurescited by 34
  10. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  11. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  12. 2026PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Trainingcited by 21
9 more
  1. 2023CLIPTER: Looking at the Bigger Picture in Scene Text Recognitioncited by 16
  2. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  3. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  4. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  5. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCRcited by 8
  6. 2026Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editionscited by 5
  7. 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Modelscited by 4
  8. 2026When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCRcited by 4
  9. 2026Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction
  2. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
  3. 2025Towards Statistical Factuality Guarantee for Large Vision-Language Models
  4. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  5. 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Models
  6. 2025Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models
  7. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptions
  8. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models