# Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

arXiv: https://arxiv.org/abs/2603.09095 · HTML: https://mlsanity.com/paper/2603.09095/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 28 of 28)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models](https://mlsanity.com/paper/2506.01062/) (2025)
- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [Glyph: Scaling Context Windows via Visual-Text Compression](https://mlsanity.com/paper/2510.17800/) (2025)
- [ARC Is a Vision Problem!](https://mlsanity.com/paper/2511.14761/) (2025)
- [Improving Language Understanding from Screenshots](https://mlsanity.com/paper/2402.14073/) (2024)
- [See the Text: From Tokenization to Visual Reading](https://mlsanity.com/paper/2510.18840/) (2025)
- [VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?](https://mlsanity.com/paper/2602.04802/) (2026)
- [Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs](https://mlsanity.com/paper/2512.08923/) (2025)
- [PixelWorld: How Far Are We from Perceiving Everything as Pixels?](https://mlsanity.com/paper/2501.19339/) (2025)
- [Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models](https://mlsanity.com/paper/2508.01678/) (2025)
- [ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models](https://mlsanity.com/paper/2505.19091/) (2025)

## Built on by (most cited first, 7 of 7)

- [OneReason Technical Report](https://mlsanity.com/paper/2606.06260/) (2026)
- [LensVLM: Selective Context Expansion for Compressed Visual Representation of Text](https://mlsanity.com/paper/2605.07019/) (2026)
- [PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.28344/) (2026)
- [Reading is not Reasoning: Bridging the Agentic Policy Gap in Vision-Text Compression](https://mlsanity.com/paper/2608.08960/) (2026)
- [When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks](https://mlsanity.com/paper/2604.27272/) (2026)
- [Which Source Wins? Task-Dependent Reliance in Vision-Language Models](https://mlsanity.com/paper/2608.17205/) (2026)
- [When MLLMs miss what humans read: diagnosing and learning to read squint text](https://mlsanity.com/paper/2610.13465/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.