mlsanityData to 14 Oct 2026

How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads

arXiv 2505.15865 ↗First posted 21 May 2025cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Despite significant advancements in Large Vision Language Models (LVLMs), a gap remains, particularly regarding their interpretability and how they locate and interpret textual information within images. In this paper, we explore various LVLMs to identify the specific heads responsible for recognizing text from images, which we term the Optical Character Recognition Head (OCR Head). Our findings regarding these heads are as follows: (1) Less Sparse: Unlike previous retrieval heads, a large number of heads are activated to extract textual information from images. (2) Qualitatively Distinct: OCR heads possess properties that differ significantly from general retrieval heads, exhibiting low similarity in their characteristics. (3) Statically Activated: The frequency of activation for these heads closely aligns with their OCR scores. We validate our findings in downstream tasks by applying Chain-of-Thought (CoT) to both OCR and conventional retrieval heads and by masking these heads. We also demonstrate that redistributing sink-token values within the OCR heads improves performance. These insights provide a deeper understanding of the internal mechanisms LVLMs employ in processing embedded textual information in images.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 35 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

23 of 35 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
11 more
  1. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  2. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  3. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  5. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  6. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  7. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 60
  8. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  9. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  10. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 35
  11. 2024How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognitioncited by 4
  2. 2026VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understandingcited by 3
  3. 2025Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877-1918)cited by 2
  4. 2026VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integrationcited by 2
  5. 2026Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Modelscited by 2
  6. 2026MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cuescited by 1
  7. 2026Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Informationcited by 1
  8. 2025Selective LoRA for Visual Tokens and Attention Headscited by 0
  9. 2026From Retrieval to Recognition:How Vision--Language Models Become OCR Specialistscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Using OCR Heads to Verbalize Image Semantics
  2. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
  3. 2025DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model
  4. 2026Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
  5. 2025See the Text: From Tokenization to Visual Reading
  6. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
  7. 2025Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual Complexity
  8. 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models