# Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context

arXiv: https://arxiv.org/abs/2302.14389 · HTML: https://mlsanity.com/paper/2302.14389/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 102 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Compositional generalization in a deep seq2seq model by separating syntax and semantics](https://mlsanity.com/paper/1904.09708/) (2019)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps](https://mlsanity.com/paper/2207.03380/) (2022)

## Built on by (most cited first, 13 of 13)

- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [Crafting Interpretable Embeddings by Asking LLMs Questions](https://mlsanity.com/paper/2405.16714/) (2024)
- [fMRI predictors based on language models of increasing complexity recover brain left lateralization](https://mlsanity.com/paper/2405.17992/) (2024)
- [Metric Learning Encoding Models: A Multivariate Framework for Interpreting Neural Representations](https://mlsanity.com/paper/2402.11608/) (2024)
- [Interpretable Next-token Prediction via the Generalized Induction Head](https://mlsanity.com/paper/2411.00066/) (2024)
- [Interpreting Brain Responses to Language with Sparse Features from Language Models](https://mlsanity.com/paper/2606.06857/) (2026)
- [Probing Brain Context-Sensitivity with Masked-Attention Generation](https://mlsanity.com/paper/2305.13863/) (2023)
- [Multipath parsing in the brain](https://mlsanity.com/paper/2401.18046/) (2024)
- [Training-Driven Representational Geometry Modularization Predicts Brain Alignment in Language Models](https://mlsanity.com/paper/2602.07539/) (2026)
- [Left-right asymmetry in predicting brain activity from LLMs' representations emerges with their formal linguistic competence](https://mlsanity.com/paper/2602.12811/) (2026)
- [Brain-LLM Alignment Tracks Training Data, Not Typology](https://mlsanity.com/paper/2605.23032/) (2026)

## Abstract (arXiv)

A fundamental question in neurolinguistics concerns the brain regions involved in syntactic and semantic processing during speech comprehension, both at the lexical (word processing) and supra-lexical levels (sentence and discourse processing). To what extent are these regions separated or intertwined? To address this question, we trained a lexical language model, Glove, and a supra-lexical language model, GPT-2, on a text corpus from which we selectively removed either syntactic or semantic information. We then assessed to what extent these information-restricted models were able to predict the time-courses of fMRI signal of humans listening to naturalistic text. We also manipulated the size of contextual information provided to GPT-2 in order to determine the windows of integration of brain regions involved in supra-lexical processing. Our analyses show that, while most brain regions involved in language are sensitive to both syntactic and semantic variables, the relative magnitudes of these effects vary a lot across these regions. Furthermore, we found an asymmetry between the left and right hemispheres, with semantic and syntactic processing being more dissociated in the left hemisphere than in the right, and the left and right hemispheres showing respectively greater sensitivity to short and long contexts. The use of information-restricted NLP models thus shed new light on the spatial organization of syntactic processing, semantic processing and compositionality.