# Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

arXiv: https://arxiv.org/abs/2106.05426 · HTML: https://mlsanity.com/paper/2106.05426/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference](https://mlsanity.com/paper/2002.04815/) (2020)

## Built on by (most cited first, 30 of 34)

- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [Do Vision and Language Encoders Represent the World Similarly?](https://mlsanity.com/paper/2401.05224/) (2024)
- [Self-supervised models of audio effectively explain human cortical responses to speech](https://mlsanity.com/paper/2205.14252/) (2022)
- [Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?](https://mlsanity.com/paper/2205.01404/) (2022)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Improving Semantic Understanding in Speech Language Models via Brain-tuning](https://mlsanity.com/paper/2410.09230/) (2024)
- [Speech language models lack important brain-relevant semantics](https://mlsanity.com/paper/2311.04664/) (2023)
- [Multimodal foundation models are better simulators of the human brain](https://mlsanity.com/paper/2208.08263/) (2022)
- [Latent Space Translation via Inverse Relative Projection](https://mlsanity.com/paper/2406.15057/) (2024)
- [Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain](https://mlsanity.com/paper/2303.14871/) (2023)
- [Bootstrapping Parallel Anchors for Relative Representations](https://mlsanity.com/paper/2303.00721/) (2023)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [Evaluating the structure of cognitive tasks with transfer learning](https://mlsanity.com/paper/2308.02408/) (2023)
- [Multi-modal brain encoding models for multi-modal stimuli](https://mlsanity.com/paper/2505.20027/) (2025)
- [Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide](https://mlsanity.com/paper/2203.05300/) (2022)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation](https://mlsanity.com/paper/2505.17099/) (2025)
- [The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models](https://mlsanity.com/paper/2310.07106/) (2023)
- [Computational Models to Study Language Processing in the Human Brain: A Survey](https://mlsanity.com/paper/2403.13368/) (2024)
- [Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models](https://mlsanity.com/paper/2510.21520/) (2025)
- [Deep Learning Models to Study Sentence Comprehension in the Human Brain](https://mlsanity.com/paper/2301.06340/) (2023)
- [Brain-like Functional Organization within Large Language Models](https://mlsanity.com/paper/2410.19542/) (2024)
- [A framework for the emergence and analysis of language in social learning agents](https://mlsanity.com/paper/2305.02632/) (2023)
- [GeRA: Label-Efficient Geometrically Regularized Alignment](https://mlsanity.com/paper/2310.00672/) (2023)
- [Do self-supervised speech and language models extract similar representations as human brain?](https://mlsanity.com/paper/2310.04645/) (2023)
- [Reprint: a randomized extrapolation based on principal components for data augmentation](https://mlsanity.com/paper/2204.12024/) (2022)

## Abstract (arXiv)

How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.