# Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models

arXiv: https://arxiv.org/abs/2210.03575 · HTML: https://mlsanity.com/paper/2210.03575/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Measuring Compositionality in Representation Learning](https://mlsanity.com/paper/1902.07181/) (2019)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [Assessing Phrasal Representation and Composition in Transformers](https://mlsanity.com/paper/2010.03763/) (2020)
- [Discovering the Compositional Structure of Vector Representations with Role Learning Networks](https://mlsanity.com/paper/1910.09113/) (2019)
- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [RNNs Implicitly Implement Tensor Product Representations](https://mlsanity.com/paper/1812.08718/) (2018)
- [Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks](https://mlsanity.com/paper/1711.00350/) (2017)
- [Investigating Robustness of Dialog Models to Popular Figurative Language Constructs](https://mlsanity.com/paper/2110.00687/) (2021)
- [Compositional Processing Emerges in Neural Networks Solving Math Problems](https://mlsanity.com/paper/2105.08961/) (2021)

## Built on by (most cited first, 9 of 9)

- [Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars](https://mlsanity.com/paper/2312.01429/) (2023)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations](https://mlsanity.com/paper/2305.14599/) (2023)
- [Divergences between Language Models and Human Brains](https://mlsanity.com/paper/2311.09308/) (2023)
- [Unified Representation for Non-compositional and Compositional Expressions](https://mlsanity.com/paper/2310.19127/) (2023)
- [Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models](https://mlsanity.com/paper/2312.11720/) (2023)
- [Investigating Idiomaticity in Word Representations](https://mlsanity.com/paper/2411.02610/) (2024)
- [Set-Theoretic Compositionality of Sentence Embeddings](https://mlsanity.com/paper/2502.20975/) (2025)

## Abstract (arXiv)

Compositionality, the phenomenon where the meaning of a phrase can be derived from its constituent parts, is a hallmark of human language. At the same time, many phrases are non-compositional, carrying a meaning beyond that of each part in isolation. Representing both of these types of phrases is critical for language understanding, but it is an open question whether modern language models (LMs) learn to do so; in this work we examine this question. We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents. We find that the representation of a parent phrase can be predicted with some accuracy given an affine transformation of its children. While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case, indicating that LMs may not accurately distinguish between compositional and non-compositional phrases. We perform a variety of analyses, shedding light on when different varieties of LMs do and do not generate compositional representations, and discuss implications for future modeling work.