# Do VLMs Align Better with Humans than LLMs during Natural Reading?

arXiv: https://arxiv.org/abs/2605.28818 · HTML: https://mlsanity.com/paper/2605.28818/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 11 of 11)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [From Language to Cognition: How LLMs Outgrow the Human Language Network](https://mlsanity.com/paper/2503.01830/) (2025)
- [Language models and brains align due to more than next-word prediction and word-level information](https://mlsanity.com/paper/2212.00596/) (2022)
- [Revealing Vision-Language Integration in the Brain with Multimodal Networks](https://mlsanity.com/paper/2406.14481/) (2024)
- [Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage](https://mlsanity.com/paper/2510.17833/) (2025)
- [Language models align with brain regions that represent concepts across modalities](https://mlsanity.com/paper/2508.11536/) (2025)
- [Improve Language Model and Brain Alignment via Associative Memory](https://mlsanity.com/paper/2505.13844/) (2025)

## Abstract (arXiv)

Large language models have become increasingly useful computational models of human language processing, but it remains open whether vision-language learning makes text representations more human-like during natural reading. We address this question by comparing matched LLM and vision-language model pairs under strictly text-only input and evaluating alignment with human brain activity (whole-cortex fMRI) and human behavior (synchronized regressive saccades). We identify a selective, rather than global, effect of vision-language training on human-model alignment. In the two within-lineage model pairs, VLMs more accurately predicted human regressive saccades, whereas VLMs and LLMs showed comparable whole-cortex fMRI alignment. However, sentence-level analyses revealed that the VLM advantage in fMRI alignment increased with the visual evocative strength of the sentences. Together, these findings provide a controlled in-silico comparison of multimodal training histories, showing that vision-language pretraining selectively improves model-human alignment via reading behavior and visually grounded content.