# Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase

arXiv: https://arxiv.org/abs/2412.04936 · HTML: https://mlsanity.com/paper/2412.04936/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 52 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 8 of 8)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [CogniVal: A Framework for Cognitive Word Embedding Evaluation](https://mlsanity.com/paper/1909.09001/) (2019)
- [Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations](https://mlsanity.com/paper/1606.00819/) (2016)

## Built on by (most cited first, 5 of 5)

- [Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models](https://mlsanity.com/paper/2510.12110/) (2025)
- [Addressing Longstanding Challenges in Cognitive Science with Language Models](https://mlsanity.com/paper/2511.00206/) (2025)
- [Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models](https://mlsanity.com/paper/2601.03798/) (2026)
- [Machine individuality: Separating genuine idiosyncrasy from response bias in large language models](https://mlsanity.com/paper/2604.16755/) (2026)
- [The "Small World of Words" German Free-Association Norms](https://mlsanity.com/paper/2604.19620/) (2026)

## Abstract (arXiv)

Semantic representations are integral to natural language processing, psycholinguistics, and artificial intelligence. Although often derived from internet text, recent years have seen a rise in the popularity of behavior-based (e.g., free associations) and brain-based (e.g., fMRI) representations, which promise improvements in our ability to measure and model human representations. We carry out the first systematic evaluation of the similarities and differences between semantic representations derived from text, behavior, and brain data. Using representational similarity analysis, we show that word vectors derived from behavior and brain data encode information that differs from their text-derived cousins. Furthermore, drawing on our psychNorms metabase, alongside an interpretability method that we call representational content analysis, we find that, in particular, behavior representations capture unique variance on certain affective, agentic, and socio-moral dimensions. We thus establish behavior as an important complement to text for capturing human representations and behavior. These results are broadly relevant to research aimed at learning human-aligned semantic representations, including work on evaluating and aligning large language models.