# Measuring Compositionality in Representation Learning

arXiv: https://arxiv.org/abs/1902.07181 · HTML: https://mlsanity.com/paper/1902.07181/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2019.

## Built on by (most cited first, 30 of 84)

- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task](https://mlsanity.com/paper/2310.09336/) (2023)
- [Towards Understanding and Improving GFlowNet Training](https://mlsanity.com/paper/2305.07170/) (2023)
- [Linear Spaces of Meanings: Compositional Structures in Vision-Language Models](https://mlsanity.com/paper/2302.14383/) (2023)
- [TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation](https://mlsanity.com/paper/2406.08656/) (2024)
- [Emergent Communication of Generalizations](https://mlsanity.com/paper/2106.02668/) (2021)
- [Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language](https://mlsanity.com/paper/2210.00482/) (2022)
- [Characterizing Intrinsic Compositionality in Transformers with Tree Projections](https://mlsanity.com/paper/2211.01288/) (2022)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?](https://mlsanity.com/paper/2203.17271/) (2022)
- [EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case](https://mlsanity.com/paper/2104.11914/) (2021)
- [Emergent Communication: Generalization and Overfitting in Lewis Games](https://mlsanity.com/paper/2209.15342/) (2022)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [Unit Testing for Concepts in Neural Networks](https://mlsanity.com/paper/2208.10244/) (2022)
- [Relational Composition in Neural Networks: A Survey and Call to Action](https://mlsanity.com/paper/2407.14662/) (2024)
- [Toward Compositional Generalization in Object-Oriented World Modeling](https://mlsanity.com/paper/2204.13661/) (2022)
- [Improving Compositional Generalization with Self-Training for Data-to-Text Generation](https://mlsanity.com/paper/2110.08467/) (2021)
- [Learning to Extrapolate: A Transductive Approach](https://mlsanity.com/paper/2304.14329/) (2023)
- [Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation](https://mlsanity.com/paper/2106.15078/) (2021)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Zero-shot Node Classification with Decomposed Graph Prototype Network](https://mlsanity.com/paper/2106.08022/) (2021)
- [On The Specialization of Neural Modules](https://mlsanity.com/paper/2409.14981/) (2024)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [A Complexity-Based Theory of Compositionality](https://mlsanity.com/paper/2410.14817/) (2024)
- [Distinguishing rule- and exemplar-based generalization in learning systems](https://mlsanity.com/paper/2110.04328/) (2021)
- [Benchmarking Compositionality with Formal Languages](https://mlsanity.com/paper/2208.08195/) (2022)
- [Greybox XAI: a Neural-Symbolic learning framework to produce interpretable predictions for image classification](https://mlsanity.com/paper/2209.14974/) (2022)
- [Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability](https://mlsanity.com/paper/2106.03982/) (2021)
- [Clustering units in neural networks: upstream vs downstream information](https://mlsanity.com/paper/2203.11815/) (2022)

## Abstract (arXiv)

Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs' learned representations. While the assessment of compositionality in languages has received significant attention in linguistics and adjacent fields, the machine learning literature lacks general-purpose tools for producing graded measurements of compositional structure in more general (e.g. vector-valued) representation spaces. We describe a procedure for evaluating compositionality by measuring how well the true representation-producing model can be approximated by a model that explicitly composes a collection of inferred representational primitives. We use the procedure to provide formal and empirical characterizations of compositional structure in a variety of settings, exploring the relationship between compositionality and learning dynamics, human judgments, representational similarity, and generalization.