# Linking artificial and human neural representations of language

arXiv: https://arxiv.org/abs/1910.01244 · HTML: https://mlsanity.com/paper/1910.01244/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 30 of 37)

- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces](https://mlsanity.com/paper/2410.09918/) (2024)
- [A Closer Look at How Fine-tuning Changes BERT](https://mlsanity.com/paper/2106.14282/) (2021)
- [Changing Answer Order Can Decrease MMLU Accuracy](https://mlsanity.com/paper/2406.19470/) (2024)
- [What if This Modified That? Syntactic Interventions via Counterfactual Embeddings](https://mlsanity.com/paper/2105.14002/) (2021)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Context Limitations Make Neural Language Models More Human-Like](https://mlsanity.com/paper/2205.11463/) (2022)
- [Language Models Trained on Media Diets Can Predict Public Opinion](https://mlsanity.com/paper/2303.16779/) (2023)
- [Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?](https://mlsanity.com/paper/2205.01404/) (2022)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Language models and brains align due to more than next-word prediction and word-level information](https://mlsanity.com/paper/2212.00596/) (2022)
- [Visio-Linguistic Brain Encoding](https://mlsanity.com/paper/2204.08261/) (2022)
- [Does injecting linguistic structure into language models lead to better alignment with brain recordings?](https://mlsanity.com/paper/2101.12608/) (2021)
- [What makes a good metric? Evaluating automatic metrics for text-to-image consistency](https://mlsanity.com/paper/2412.13989/) (2024)
- [Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects](https://mlsanity.com/paper/2110.06078/) (2021)
- [Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords](https://mlsanity.com/paper/2109.11491/) (2021)
- [Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain](https://mlsanity.com/paper/2303.14871/) (2023)
- [MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models](https://mlsanity.com/paper/2403.01116/) (2024)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?](https://mlsanity.com/paper/2310.01854/) (2023)
- [On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations](https://mlsanity.com/paper/2101.11492/) (2021)
- [Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide](https://mlsanity.com/paper/2203.05300/) (2022)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [Sometimes We Want Translationese](https://mlsanity.com/paper/2104.07623/) (2021)
- [Probing Semantic Grounding in Language Models of Code with Representational Similarity Analysis](https://mlsanity.com/paper/2207.07706/) (2022)
- [Divergences between Language Models and Human Brains](https://mlsanity.com/paper/2311.09308/) (2023)
- [Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models](https://mlsanity.com/paper/2104.08410/) (2021)
- [How Many Bytes Can You Take Out Of Brain-To-Text Decoding?](https://mlsanity.com/paper/2405.14055/) (2024)

## Abstract (arXiv)

What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance. We find that none of the sentence encoding tasks tested yield significant increases in brain decoding performance. Through further task ablations and representational analyses, we find that tasks which produce syntax-light representations yield significant improvements in brain decoding performance. Our results constrain the space of NLU models that could best account for human neural representations of language, but also suggest limits on the possibility of decoding fine-grained syntactic information from fMRI human neuroimaging.