# Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

arXiv: https://arxiv.org/abs/2205.01404 · HTML: https://mlsanity.com/paper/2205.01404/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 56 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Inducing brain-relevant bias in natural language processing models](https://mlsanity.com/paper/1911.03268/) (2019)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- [Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses](https://mlsanity.com/paper/2106.05426/) (2021)
- [CogniVal: A Framework for Cognitive Word Embedding Evaluation](https://mlsanity.com/paper/1909.09001/) (2019)
- [Interpreting Encoding and Decoding Models](https://mlsanity.com/paper/1812.00278/) (2018)
- [Visio-Linguistic Brain Encoding](https://mlsanity.com/paper/2204.08261/) (2022)
- [Relating Simple Sentence Representations in Deep Neural Networks and the Brain](https://mlsanity.com/paper/1906.11861/) (2019)
- [Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects](https://mlsanity.com/paper/2110.06078/) (2021)
- [Modeling Task Effects on Meaning Representation in the Brain via Zero-Shot MEG Prediction](https://mlsanity.com/paper/2009.08424/) (2020)
- [fMRI Semantic Category Decoding using Linguistic Encoding of Word Embeddings](https://mlsanity.com/paper/1806.05177/) (2018)
- [Cross-view Brain Decoding](https://mlsanity.com/paper/2204.09564/) (2022)

## Built on by (most cited first, 27 of 27)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [Understanding LLMs: A Comprehensive Overview from Training to Inference](https://mlsanity.com/paper/2401.02038/) (2024)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Training language models to summarize narratives improves brain alignment](https://mlsanity.com/paper/2212.10898/) (2022)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Improving Semantic Understanding in Speech Language Models via Brain-tuning](https://mlsanity.com/paper/2410.09230/) (2024)
- [Speech language models lack important brain-relevant semantics](https://mlsanity.com/paper/2311.04664/) (2023)
- [What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores](https://mlsanity.com/paper/2406.01538/) (2024)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [Evaluating the structure of cognitive tasks with transfer learning](https://mlsanity.com/paper/2308.02408/) (2023)
- [Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?](https://mlsanity.com/paper/2310.01854/) (2023)
- [Multi-modal brain encoding models for multi-modal stimuli](https://mlsanity.com/paper/2505.20027/) (2025)
- [Computational Models to Study Language Processing in the Human Brain: A Survey](https://mlsanity.com/paper/2403.13368/) (2024)
- [Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models](https://mlsanity.com/paper/2404.19364/) (2024)
- [Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)](https://mlsanity.com/paper/2505.20029/) (2025)
- [Language models align with brain regions that represent concepts across modalities](https://mlsanity.com/paper/2508.11536/) (2025)
- [Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models](https://mlsanity.com/paper/2510.21520/) (2025)
- [Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language](https://mlsanity.com/paper/2310.04460/) (2023)
- [Divergences between Language Models and Human Brains](https://mlsanity.com/paper/2311.09308/) (2023)
- [Vision-Language Models Align with Human Neural Representations in Concept Processing](https://mlsanity.com/paper/2407.17914/) (2024)
- [Syntactic Structure Processing in the Brain while Listening](https://mlsanity.com/paper/2302.08589/) (2023)
- [Neural Correlates of Language Models Are Specific to Human Language](https://mlsanity.com/paper/2510.03156/) (2025)
- [Ensemble of Task-Specific Language Models for Brain Encoding](https://mlsanity.com/paper/2310.15720/) (2023)
- [Linguistic properties and model scale in brain encoding: from small to compressed language models](https://mlsanity.com/paper/2602.07547/) (2026)
- [Cognitive Expert Language Models Better Align with the Corresponding Brain Systems](https://mlsanity.com/paper/2609.36239/) (2026)

## Abstract (arXiv)

Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporo-parieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.