mlsanityData to 14 Oct 2026

Language models align with brain regions that represent concepts across modalities

arXiv 2508.11536 ↗First posted 15 Aug 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM--brain alignment and two neural metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) a novel measure of meaning consistency across input modalities, which quantifies how consistently a brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the signal better in more meaning-consistent areas of the brain, even when these areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 86 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

35 of 86 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
23 more
  1. 2023Getting aligned on representational alignmentcited by 143
  2. 2022Linearly Mapping from Image to Text Spacecited by 142
  3. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  4. 2015Generic decoding of seen and imagined objects using hierarchical visual featurescited by 120
  5. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 119
  6. 2022Meaning without reference in large language modelscited by 76
  7. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  8. 2023Scaling laws for language encoding models in fMRIcited by 68
  9. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  10. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 40
  11. 2022Joint processing of linguistic properties in brains and language modelscited by 37
  12. 2022Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?cited by 27
  13. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27
  14. 2022Training language models to summarize narratives improves brain alignmentcited by 26
  15. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 25
  16. 2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 21
  17. 2023Speech language models lack important brain-relevant semanticscited by 19
  18. 2022Language models and brains align due to more than next-word prediction and word-level informationcited by 14
  19. 2022Visio-Linguistic Brain Encodingcited by 13
  20. 2024Universal dimensions of visual representationcited by 12
  21. 2023Mapping Brains with Language Models: A Surveycited by 8
  22. 2023Structural Similarities Between Language Models and Neural Response Measurementscited by 6
  23. 2025Experiential Semantic Information and Brain Alignment: Are Multimodal Models Better than Language Models?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantagecited by 5
  2. 2026Do VLMs Align Better with Humans than LLMs during Natural Reading?cited by 0
  3. 2026Validating Causal Abstraction Metrics on Simulated Complex Systemscited by 0
  4. 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  2. 2026Cross-lingual brain-language model alignment is robust but challenges hierarchical and computational accounts
  3. 2026Do Language Models Align with Brains? Prediction Scores Are Not Enough
  4. 2026Cognitive Expert Language Models Better Align with the Corresponding Brain Systems
  5. 2021Does injecting linguistic structure into language models lead to better alignment with brain recordings?
  6. 2026Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignment
  7. 2026Linguistic properties and model scale in brain encoding: from small to compressed language models
  8. 2026When Language Models Lose Their Mind: The Consequences of Brain Misalignment