# A Psycholinguistic Analysis of BERT's Representations of Compounds

arXiv: https://arxiv.org/abs/2302.07232 · HTML: https://mlsanity.com/paper/2302.07232/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition](https://mlsanity.com/paper/1902.10618/) (2019)
- [A comparative evaluation and analysis of three generations of Distributional Semantic Models](https://mlsanity.com/paper/2105.09825/) (2021)
- [What do you mean, BERT? Assessing BERT as a Distributional Semantics Model](https://mlsanity.com/paper/1911.05758/) (2019)
- [Will it Unblend?](https://mlsanity.com/paper/2009.09123/) (2020)
- [Don't Blame Distributional Semantics if it can't do Entailment](https://mlsanity.com/paper/1905.07356/) (2019)

## Built on by (most cited first, 7 of 7)

- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Can AI mimic the human ability to define neologisms?](https://mlsanity.com/paper/2502.14900/) (2025)
- [How do Transformer Embeddings Represent Compositions? A Functional Analysis](https://mlsanity.com/paper/2506.00914/) (2025)
- [Animate, or Inanimate, That is the Question for Large Language Models](https://mlsanity.com/paper/2408.06332/) (2024)
- [Probing BERT for German Compound Semantics](https://mlsanity.com/paper/2505.14130/) (2025)
- [The aftermath of compounds: Investigating Compounds and their Semantic Representations](https://mlsanity.com/paper/2510.27477/) (2025)

## Abstract (arXiv)

This work studies the semantic representations learned by BERT for compounds, that is, expressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions when dealing with expressions (e.g., sunlight) whose overall meaning depends -- to a various extent -- on the semantics of the constituent words (sun, light). We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD; quantifying the weight of each constituent toward the compound meaning) and semantic transparency (ST; evaluating the extent to which the compound meaning is recoverable from the constituents' semantics). We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations, and that LMD is overall more predictable than ST. Contrary to the results reported for 'standard' words, higher, more contextualized layers are the best at representing compound meaning. These findings shed new light on the abilities of BERT in dealing with fine-grained semantic phenomena. Moreover, they can provide insights into how speakers represent compounds.