Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
Abstract (arXiv)
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 29 references that are in the map. First posted on arXiv 26 Feb 2026.
Builds on
13 of 29 references.
Closest ancestors · 1
Same direction, newest first
- 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads21 May
Other references · 12
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
- 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
- 2023Teaching CLIP to Count to Tencited by 194
- 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
- 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
- 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
- 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lenscited by 9
- 2025Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIPcited by 8
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists
- 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR
- 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
- 2026What Can Low Resource Languages Learn From Each Other?
- 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens
- 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models
- 2025Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models
- 2026Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models