# Language models and brains align due to more than next-word prediction and word-level information

arXiv: https://arxiv.org/abs/2212.00596 · HTML: https://mlsanity.com/paper/2212.00596/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 40 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 8 of 8)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [UnNatural Language Inference](https://mlsanity.com/paper/2101.00010/) (2020)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- [Sorting through the noise: Testing robustness of information processing in pre-trained language models](https://mlsanity.com/paper/2109.12393/) (2021)
- [When classifying grammatical role, BERT doesn't care about word order... except when it matters](https://mlsanity.com/paper/2203.06204/) (2022)
- [Does injecting linguistic structure into language models lead to better alignment with brain recordings?](https://mlsanity.com/paper/2101.12608/) (2021)
- [Same Cause; Different Effects in the Brain](https://mlsanity.com/paper/2202.10376/) (2022)

## Built on by (most cited first, 14 of 14)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Training language models to summarize narratives improves brain alignment](https://mlsanity.com/paper/2212.10898/) (2022)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage](https://mlsanity.com/paper/2510.17833/) (2025)
- [Language models align with brain regions that represent concepts across modalities](https://mlsanity.com/paper/2508.11536/) (2025)
- [Fine-grained Analysis of Brain-LLM Alignment through Input Attribution](https://mlsanity.com/paper/2510.12355/) (2025)
- [Syntactic Structure Processing in the Brain while Listening](https://mlsanity.com/paper/2302.08589/) (2023)
- [Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report)](https://mlsanity.com/paper/2510.06235/) (2025)
- [Interpreting Brain Responses to Language with Sparse Features from Language Models](https://mlsanity.com/paper/2606.06857/) (2026)
- [Brain-LLM Alignment Tracks Training Data, Not Typology](https://mlsanity.com/paper/2605.23032/) (2026)
- [Do VLMs Align Better with Humans than LLMs during Natural Reading?](https://mlsanity.com/paper/2605.28818/) (2026)

## Abstract (arXiv)

Pretrained language models have been shown to significantly predict brain recordings of people comprehending language. Recent work suggests that the prediction of the next word is a key mechanism that contributes to this alignment. What is not yet understood is whether prediction of the next word is necessary for this observed alignment or simply sufficient, and whether there are other shared mechanisms or information that are similarly important. In this work, we take a step towards understanding the reasons for brain alignment via two simple perturbations in popular pretrained language models. These perturbations help us design contrasts that can control for different types of information. By contrasting the brain alignment of these differently perturbed models, we show that improvements in alignment with brain recordings are due to more than improvements in next-word prediction and word-level information.