# Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps

arXiv: https://arxiv.org/abs/2207.03380 · HTML: https://mlsanity.com/paper/2207.03380/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 34 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects](https://mlsanity.com/paper/2110.06078/) (2021)
- [Modeling Task Effects on Meaning Representation in the Brain via Zero-Shot MEG Prediction](https://mlsanity.com/paper/2009.08424/) (2020)

## Built on by (most cited first, 26 of 26)

- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [From Language to Cognition: How LLMs Outgrow the Human Language Network](https://mlsanity.com/paper/2503.01830/) (2025)
- [Training language models to summarize narratives improves brain alignment](https://mlsanity.com/paper/2212.10898/) (2022)
- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context](https://mlsanity.com/paper/2302.14389/) (2023)
- [What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores](https://mlsanity.com/paper/2406.01538/) (2024)
- [Probing the limit of hydrologic predictability with the Transformer network](https://mlsanity.com/paper/2306.12384/) (2023)
- [MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models](https://mlsanity.com/paper/2403.01116/) (2024)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [fMRI predictors based on language models of increasing complexity recover brain left lateralization](https://mlsanity.com/paper/2405.17992/) (2024)
- [Computational Models to Study Language Processing in the Human Brain: A Survey](https://mlsanity.com/paper/2403.13368/) (2024)
- [Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models](https://mlsanity.com/paper/2404.19364/) (2024)
- [Divergences between Language Models and Human Brains](https://mlsanity.com/paper/2311.09308/) (2023)
- [Metric Learning Encoding Models: A Multivariate Framework for Interpreting Neural Representations](https://mlsanity.com/paper/2402.11608/) (2024)
- [Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network](https://mlsanity.com/paper/2406.15109/) (2024)
- [Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography](https://mlsanity.com/paper/2605.23035/) (2026)
- [Scaling up ridge regression for brain encoding in a massive individual fMRI dataset](https://mlsanity.com/paper/2403.19421/) (2024)
- [No Such Thing as a General Learner: Language models and their dual optimization](https://mlsanity.com/paper/2408.09544/) (2024)
- [Brain-like emergent properties in deep networks: impact of network architecture, datasets and training](https://mlsanity.com/paper/2411.16326/) (2024)
- [Neural Correlates of Language Models Are Specific to Human Language](https://mlsanity.com/paper/2510.03156/) (2025)
- [Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweighting](https://mlsanity.com/paper/2510.18745/) (2025)
- [Probing Brain Context-Sensitivity with Masked-Attention Generation](https://mlsanity.com/paper/2305.13863/) (2023)
- [Training Compute-Optimal Vision Transformers for Brain Encoding](https://mlsanity.com/paper/2410.19810/) (2024)
- [Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility](https://mlsanity.com/paper/2505.16277/) (2025)
- [Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequences](https://mlsanity.com/paper/2604.15503/) (2026)

## Abstract (arXiv)

Neural Language Models (NLMs) have made tremendous advances during the last years, achieving impressive performance on various linguistic tasks. Capitalizing on this, studies in neuroscience have started to use NLMs to study neural activity in the human brain during language processing. However, many questions remain unanswered regarding which factors determine the ability of a neural language model to capture brain activity (aka its 'brain score'). Here, we make first steps in this direction and examine the impact of test loss, training corpus and model architecture (comparing GloVe, LSTM, GPT-2 and BERT), on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that (1) untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words, with the untrained LSTM outperforming the transformerbased models, being less impacted by the effect of context; (2) that training NLP models improves brain scores in the same brain regions irrespective of the model's architecture; (3) that Perplexity (test loss) is not a good predictor of brain score; (4) that training data have a strong influence on the outcome and, notably, that off-the-shelf models may lack statistical power to detect brain activations. Overall, we outline the impact of modeltraining choices, and suggest good practices for future studies aiming at explaining the human language system using neural language models.