mlsanityData to 14 Oct 2026

Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding

arXiv 2510.16870 ↗First posted 19 Oct 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While brain-inspired artificial intelligence(AI) has demonstrated promising results, current understanding of the parallels between artificial neural networks (ANNs) and human brain processing remains limited: (1) unimodal ANN studies fail to capture the brain's inherent multimodal processing capabilities, and (2) multimodal ANN research primarily focuses on high-level model outputs, neglecting the crucial role of individual neurons. To address these limitations, we propose a novel neuron-level analysis framework that investigates the multimodal information processing mechanisms in vision-language models (VLMs) through the lens of human brain activity. Our approach uniquely combines fine-grained artificial neuron (AN) analysis with fMRI-based voxel encoding to examine two architecturally distinct VLMs: CLIP and METER. Our analysis reveals four key findings: (1) ANs successfully predict biological neurons (BNs) activities across multiple functional networks (including language, vision, attention, and default mode), demonstrating shared representational mechanisms; (2) Both ANs and BNs demonstrate functional redundancy through overlapping neural representations, mirroring the brain's fault-tolerant and collaborative information processing mechanisms; (3) ANs exhibit polarity patterns that parallel the BNs, with oppositely activated BNs showing mirrored activation trends across VLM layers, reflecting the complexity and bidirectional nature of neural information processing; (4) The architectures of CLIP and METER drive distinct BNs: CLIP's independent branches show modality-specific specialization, whereas METER's cross-modal design yields unified cross-modal activation, highlighting the architecture's influence on ANN brain-like properties. These results provide compelling evidence for brain-like hierarchical processing in VLMs at the neuronal level.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 43 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

17 of 43 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  10. 2018Cognitive computational neurosciencecited by 43
  11. 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 32
  12. 2024Revealing Vision-Language Integration in the Brain with Multimodal Networkscited by 11
5 more
  1. 2022Multimodal foundation models are better simulators of the human braincited by 10
  2. 2023Coupling Artificial Neurons in BERT and Biological Neurons in the Human Braincited by 9
  3. 2025Multi-modal brain encoding models for multi-modal stimulicited by 8
  4. 2024Brain-like Functional Organization within Large Language Modelscited by 3
  5. 2025Neural encoding with affine feature response transformscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
  2. 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models
  3. 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
  4. 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
  5. 2025Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
  6. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  7. 2026Interpreting Brain Responses to Language with Sparse Features from Language Models
  8. 2026BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language