mlsanityData to 14 Oct 2026

From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists

arXiv 2609.21543 ↗First posted 18 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Does a general vision--language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677-0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17-20 of the top 20 heads per task with all-head rank correlations of 0.874-0.942, while redistributing their functional and causal strengths.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 18 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

18 of 18 references.

References · 18

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  4. 2021OCR-free Document Understanding Transformercited by 407
  5. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  6. 2019Image-based table recognition: data, model, and evaluationcited by 237
  7. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  8. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
  9. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  10. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
  11. 2024Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matchingcited by 41
  12. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 30
6 more
  1. 2026GLM-OCR Technical Reportcited by 26
  2. 2024Optimal ablation for interpretabilitycited by 25
  3. 2026PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Trainingcited by 21
  4. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Headscited by 9
  5. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Headscited by 4
  6. 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
  2. 2026Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
  3. 2025DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model
  4. 2026OCR-Agent: Agentic OCR with Capability and Memory Reflection
  5. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR
  6. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
  7. 2026Using OCR Heads to Verbalize Image Semantics
  8. 2026Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning