mlsanityData to 7 Oct 2026

Do VLMs Align Better with Humans than LLMs during Natural Reading?

arXiv 2605.28818 ↗First posted 27 May 2026cs.CL q-bio.NCcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large language models have become increasingly useful computational models of human language processing, but it remains open whether vision-language learning makes text representations more human-like during natural reading. We address this question by comparing matched LLM and vision-language model pairs under strictly text-only input and evaluating alignment with human brain activity (whole-cortex fMRI) and human behavior (synchronized regressive saccades). We identify a selective, rather than global, effect of vision-language training on human-model alignment. In the two within-lineage model pairs, VLMs more accurately predicted human regressive saccades, whereas VLMs and LLMs showed comparable whole-cortex fMRI alignment. However, sentence-level analyses revealed that the VLM advantage in fMRI alignment increased with the visual evocative strength of the sentences. Together, these findings provide a controlled in-silico comparison of multimodal training histories, showing that vision-language pretraining selectively improves model-human alignment via reading behavior and visually grounded content.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 37 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

11 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep

Other references · 10

  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  2. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  3. 2023Scaling laws for language encoding models in fMRIcited by 68
  4. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 39
  5. 2025From Language to Cognition: How LLMs Outgrow the Human Language Networkcited by 30
  6. 2022Language models and brains align due to more than next-word prediction and word-level informationcited by 14
  7. 2024Revealing Vision-Language Integration in the Brain with Multimodal Networkscited by 11
  8. 2025Language models align with brain regions that represent concepts across modalitiescited by 4
  9. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantagecited by 4
  10. 2025Improve Language Model and Brain Alignment via Associative Memorycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  2. 2025Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
  3. 2026Do Models See in Line with Human Vision? Probing the Correspondence Between LVLM Representations and EEG Signals
  4. 2024Do Large Language Models Mirror Cognitive Language Processing?
  5. 2026Brain-LLM Alignment Tracks Training Data, Not Typology
  6. 2026Linguistic properties and model scale in brain encoding: from small to compressed language models
  7. 2026Cognitive Expert Language Models Better Align with the Corresponding Brain Systems
  8. 2022Training language models to summarize narratives improves brain alignment