mlsanityData to 14 Oct 2026

Vision-Language Models Align with Human Neural Representations in Concept Processing

arXiv 2407.17914 ↗First posted 25 Jul 2024cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent studies suggest that transformer-based vision-language models (VLMs) capture the multimodality of concept processing in the human brain. However, a systematic evaluation exploring different types of VLM architectures and the role played by visual and textual context is still lacking. Here, we analyse multiple VLMs employing different strategies to integrate visual and textual modalities, along with language-only counterparts. We measure the alignment between concept representations by models and existing (fMRI) brain responses to concept words presented in two experimental conditions, where either visual (pictures) or textual (sentences) context is provided. Our results reveal that VLMs outperform the language-only counterparts in both experimental conditions. However, controlled ablation studies show that only for some VLMs, such as LXMERT and IDEFICS2, brain alignment stems from genuinely learning more human-like concepts during pretraining, while others are highly sensitive to the context provided at inference. Additionally, we find that vision-language encoders are more brain-aligned than more recent, generative VLMs. Altogether, our study shows that VLMs align with human neural representations in concept processing, while highlighting differences among architectures. We open-source code and materials to reproduce our experiments at: https://github.com/dmg-illc/vl-concept-processing.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 53 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

19 of 53 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  9. 2024Generative Representational Instruction Tuningcited by 233
  10. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  11. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  12. 2022Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?cited by 27
7 more
  1. 2021Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Modelscited by 18
  2. 2021Probing artificial neural networks: insights from neurosciencecited by 17
  3. 2023Visual Grounding Helps Learn Word Meanings in Low-Data Regimescited by 17
  4. 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
  5. 2023Is Multimodal Vision Supervision Beneficial to Language?cited by 6
  6. 2024Lexicon-Level Contrastive Visual-Grounding Improves Language Modelingcited by 6
  7. 2024Modality-Agnostic fMRI Decoding of Vision and Languagecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Modelscited by 0
  2. 2026Mechanistically Interpretable Neural Encoding Reveals Fine-Grained Functional Selectivity in Human Visual Cortexcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do VLMs Align Better with Humans than LLMs during Natural Reading?
  2. 2025Language models align with brain regions that represent concepts across modalities
  3. 2026Do Models See in Line with Human Vision? Probing the Correspondence Between LVLM Representations and EEG Signals
  4. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set
  5. 2025Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
  6. 2025Visual Language Models show widespread visual deficits on neuropsychological tests
  7. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Study
  8. 2024Human-like object concept representations emerge naturally in multimodal large language models