# Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language

arXiv: https://arxiv.org/abs/2310.04460 · HTML: https://mlsanity.com/paper/2310.04460/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction](https://mlsanity.com/paper/2104.07650/) (2021)
- [Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities](https://mlsanity.com/paper/2305.17214/) (2023)
- [Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?](https://mlsanity.com/paper/2205.01404/) (2022)
- [Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects](https://mlsanity.com/paper/2110.06078/) (2021)
- [Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?](https://mlsanity.com/paper/2310.01854/) (2023)
- [Mapping Between fMRI Responses to Movies and their Natural Language Annotations](https://mlsanity.com/paper/1610.03914/) (2016)

## Built on by (most cited first, 3 of 3)

- [Computational Models to Study Language Processing in the Human Brain: A Survey](https://mlsanity.com/paper/2403.13368/) (2024)
- [Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)](https://mlsanity.com/paper/2505.20029/) (2025)
- [Divergences between Language Models and Human Brains](https://mlsanity.com/paper/2311.09308/) (2023)

## Abstract (arXiv)

To understand the algorithm that supports the human brain's language representation, previous research has attempted to predict neural responses to linguistic stimuli using embeddings generated by artificial neural networks (ANNs), a process known as neural encoding. However, most of these studies have focused on probing neural representations of Germanic languages, such as English, with unsupervised ANNs. In this paper, we propose to bridge the gap between human brain and supervised ANN representations of the Chinese language. Specifically, we investigate how task tuning influences a pretained Transformer for neural encoding and which tasks lead to the best encoding performances. We generate supervised representations on eight Natural Language Understanding (NLU) tasks using prompt-tuning, a technique that is seldom explored in neural encoding for language. We demonstrate that prompt-tuning yields representations that better predict neural responses to Chinese stimuli than traditional fine-tuning on four tasks. Furthermore, we discover that tasks that require a fine-grained processing of concepts and entities lead to representations that are most predictive of brain activation patterns. Additionally, we reveal that the proportion of tuned parameters highly influences the neural encoding performance of fine-tuned models. Overall, our experimental findings could help us better understand the relationship between supervised artificial and brain language representations.