# Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

arXiv: https://arxiv.org/abs/1905.11833 · HTML: https://mlsanity.com/paper/1905.11833/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 30 of 128)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Incorporating Residual and Normalization Layers into Analysis of Masked Language Models](https://mlsanity.com/paper/2109.07152/) (2021)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [Self-supervised models of audio effectively explain human cortical responses to speech](https://mlsanity.com/paper/2205.14252/) (2022)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Emergent Modularity in Pre-trained Transformers](https://mlsanity.com/paper/2305.18390/) (2023)
- [The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes](https://mlsanity.com/paper/2301.03198/) (2023)
- [Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses](https://mlsanity.com/paper/2106.05426/) (2021)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language](https://mlsanity.com/paper/2311.04930/) (2023)
- [Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain](https://mlsanity.com/paper/2401.17671/) (2024)
- [Multilingual Language Models Predict Human Reading Behavior](https://mlsanity.com/paper/2104.05433/) (2021)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?](https://mlsanity.com/paper/2205.01404/) (2022)
- [Do Vision and Language Models Share Concepts? A Vector Space Alignment Study](https://mlsanity.com/paper/2302.06555/) (2023)
- [Training language models to summarize narratives improves brain alignment](https://mlsanity.com/paper/2212.10898/) (2022)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Improving Semantic Understanding in Speech Language Models via Brain-tuning](https://mlsanity.com/paper/2410.09230/) (2024)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Speech language models lack important brain-relevant semantics](https://mlsanity.com/paper/2311.04664/) (2023)
- [TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction](https://mlsanity.com/paper/2507.22229/) (2025)
- [Large language models can segment narrative events similarly to humans](https://mlsanity.com/paper/2301.10297/) (2023)
- [Language models and brains align due to more than next-word prediction and word-level information](https://mlsanity.com/paper/2212.00596/) (2022)
- [MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding](https://mlsanity.com/paper/2502.15786/) (2025)

## Abstract (arXiv)

Neural networks models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. This has generated a lot of research interest in interpreting the representations learned by these networks. We propose here a novel interpretation approach that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transformer-XL. We study how their representations differ across layer depth, context length, and attention type. Our results reveal differences in the context-related representations across these models. Further, in the transformer models, we find an interaction between layer depth and context length, and between layer depth and attention type. We finally hypothesize that altering BERT to better align with brain recordings would enable it to also better understand language. Probing the altered BERT using syntactic NLP tasks reveals that the model with increased brain-alignment outperforms the original model. Cognitive neuroscientists have already begun using NLP networks to study the brain, and this work closes the loop to allow the interaction between NLP and cognitive neuroscience to be a true cross-pollination.