Abstraction Induces the Brain Alignment of Language and Speech Models
Not on the map. Why?
Abstract (arXiv)
Research has repeatedly demonstrated that intermediate hidden states extracted from large language models and speech audio models predict measured brain response to natural language stimuli. Yet, very little is known about the representation properties that enable this high prediction performance. Why is it the intermediate layers, and not the output layers, that are most effective for this unique and highly general transfer task? We give evidence that the correspondence between speech and language models and the brain derives from shared meaning abstraction and not their next-word prediction properties. In particular, models construct higher-order linguistic features in their middle layers, cued by a peak in the layerwise intrinsic dimension, a measure of feature complexity. We show that a layer's intrinsic dimension strongly predicts how well it explains fMRI and ECoG signals; that the relation between intrinsic dimension and brain predictivity arises over model pre-training; and finetuning models to better predict the brain causally increases both representations' intrinsic dimension and their semantic content. Results suggest that semantic richness, high intrinsic dimension, and brain predictivity mirror each other, and that the key driver of model-brain similarity is rich meaning abstraction of the inputs, where language modeling is a task complex enough (but perhaps not the only) to require it.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 69 references that are in the map. First posted on arXiv 3 Feb 2026.
Builds on
35 of 69 references.
References · 35
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
- 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
- 2019Intrinsic dimension of data representations in deep neural networkscited by 276
- 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
23 more
- 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
- 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
- 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
- 2023The geometry of hidden representations of large transformer modelscited by 108
- 2023Scaling laws for language encoding models in fMRIcited by 68
- 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
- 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
- 2022Self-supervised models of audio effectively explain human cortical responses to speechcited by 37
- 2022Joint processing of linguistic properties in brains and language modelscited by 37
- 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
- 2024Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Braincited by 32
- 2025From Language to Cognition: How LLMs Outgrow the Human Language Networkcited by 30
- 2023Bridging Information-Theoretic and Geometric Compression in Language Modelscited by 26
- 2024Improving Semantic Understanding in Speech Language Models via Brain-tuningcited by 20
- 2023Speech language models lack important brain-relevant semanticscited by 19
- 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
- 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
- 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
- 2025A Survey of Dimension Estimation Methodscited by 11
- 2025A quantitative analysis of semantic information in deep representations of text and imagescited by 7
- 2025BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Languagecited by 6
- 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
- 2026Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representationscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Cross-lingual brain-language model alignment is robust but challenges hierarchical and computational accounts
- 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage
- 2026Linguistic properties and model scale in brain encoding: from small to compressed language models
- 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
- 2026Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography
- 2022Language models and brains align due to more than next-word prediction and word-level information
- 2026Do Language Models Align with Brains? Prediction Scores Are Not Enough
- 2025Language models align with brain regions that represent concepts across modalities