mlsanityData to 14 Oct 2026

Using OCR Heads to Verbalize Image Semantics

arXiv 2609.18823 ↗First posted 16 Sep 2026cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens. For example, pointing these heads at an image token containing the word "bike" causes Qwen3-VL-8B to output "bike," but pointing them at a bird wing causes the model to output the token "feathers." We collapse these heads' attention weights into a single verbalization lens transformation that reveals interpretable semantic features in hidden states across all layers. When combined with projection to vocabulary space, we can obtain interpretable labels starting from layer 0, showing that image representations are in fact aligned with language in early layers. We find that we can also use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver in a naturalistic image, providing causal evidence that this subspace is useful for more than just OCR. Our results are an example of how the study of specific mechanisms can shed light on broader interpretability problems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

49 of 62 references.

References · 49

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2023Vision Transformers Need Registerscited by 871
  6. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  8. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  10. 2024The Platonic Representation Hypothesiscited by 382
  11. 2023Function Vectors in Large Language Modelscited by 308
  12. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
37 more
  1. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  2. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  3. 2022Linearly Mapping from Image to Text Spacecited by 142
  4. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
  5. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  6. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  7. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  8. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  9. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  10. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  11. 2022Disentangling visual and written concepts in CLIPcited by 58
  12. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  13. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  14. 2025Vision Transformers Don't Need Trained Registerscited by 41
  15. 2024Backward Lens: Projecting Language Model Gradients into the Vocabulary Spacecited by 32
  16. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  17. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  18. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMscited by 25
  19. 2025The Dual-Route Model of Inductioncited by 24
  20. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  21. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 23
  22. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  23. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  24. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  25. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  26. 2023Interpreting and Controlling Vision Foundation Models via Text Explanationscited by 11
  27. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCRcited by 8
  28. 2025LLMs Process Lists With General Filter Headscited by 7
  29. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 7
  30. 2026Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editionscited by 5
  31. 2026Gaze Heads: How VLMs Look at What They Describecited by 5
  32. 2025Vector Arithmetic in Concept and Token Subspacescited by 4
  33. 2025Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Modelscited by 3
  34. 2026Representations of Text and Images Align From Layer Onecited by 3
  35. 2026Decomposing Query-Key Feature Interactions Using Contrastive Covariancescited by 3
  36. 2025Reading Images Like Texts: Sequential Image Understanding in Vision-Language Modelscited by 2
  37. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflictscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads
  2. 2026From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists
  3. 2026Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
  4. 2025DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model
  5. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
  6. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation
  7. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  8. 2026OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding