# Assessing Composition in Sentence Vector Representations

arXiv: https://arxiv.org/abs/1809.03992 · HTML: https://mlsanity.com/paper/1809.03992/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2018.

## Built on by (most cited first, 26 of 26)

- [CLiMP: A Benchmark for Chinese Language Model Evaluation](https://mlsanity.com/paper/2101.11131/) (2021)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Refining Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/2104.09635/) (2021)
- [BERTnesia: Investigating the capture and forgetting of knowledge in BERT](https://mlsanity.com/paper/2106.02902/) (2021)
- [Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding](https://mlsanity.com/paper/2109.04947/) (2021)
- [Cheetah: Natural Language Generation for 517 African Languages](https://mlsanity.com/paper/2401.01053/) (2024)
- [Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5](https://mlsanity.com/paper/2410.11627/) (2024)
- [Scope Ambiguities in Large Language Models](https://mlsanity.com/paper/2404.04332/) (2024)
- [The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models](https://mlsanity.com/paper/2312.00960/) (2023)
- [Generative linguistics contribution to artificial intelligence: Where this contribution lies?](https://mlsanity.com/paper/2410.20221/) (2024)
- [What fifty-one years of Linguistics and Artificial Intelligence research tell us about their correlation: A scientometric analysis](https://mlsanity.com/paper/2411.19858/) (2024)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [On the Interplay Between Fine-tuning and Composition in Transformers](https://mlsanity.com/paper/2105.14668/) (2021)
- [OOD-Probe: A Neural Interpretation of Out-of-Domain Generalization](https://mlsanity.com/paper/2208.12352/) (2022)
- [Interpretability of Language Models via Task Spaces](https://mlsanity.com/paper/2406.06441/) (2024)
- [Understanding Subword Compositionality of Large Language Models](https://mlsanity.com/paper/2508.17953/) (2025)
- [Variation and generality in encoding of syntactic anomaly information in sentence embeddings](https://mlsanity.com/paper/2111.06644/) (2021)
- [Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets](https://mlsanity.com/paper/2106.03631/) (2021)
- [TagRec++: Hierarchical Label Aware Attention Network for Question Categorization](https://mlsanity.com/paper/2208.05152/) (2022)
- ["No, they did not": Dialogue response dynamics in pre-trained language models](https://mlsanity.com/paper/2210.02526/) (2022)
- [Sequential Compositional Generalization in Multimodal Models](https://mlsanity.com/paper/2404.12013/) (2024)
- [Distributed neural encoding of binding to thematic roles](https://mlsanity.com/paper/2110.12342/) (2021)
- [Systematic word meta-sense extension](https://mlsanity.com/paper/2311.13029/) (2023)
- [Do Large Language Models know who did what to whom?](https://mlsanity.com/paper/2504.16884/) (2025)
- [Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence](https://mlsanity.com/paper/2507.22197/) (2025)
- [Principles of Concept Representation in Sentence Encoders](https://mlsanity.com/paper/2606.06994/) (2026)

## Abstract (arXiv)

An important component of achieving language understanding is mastering the composition of sentence meaning, but an immediate challenge to solving this problem is the opacity of sentence vector representations produced by current neural sentence composition models. We present a method to address this challenge, developing tasks that directly target compositional meaning information in sentence vector representations with a high degree of precision and control. To enable the creation of these controlled tasks, we introduce a specialized sentence generation system that produces large, annotated sentence sets meeting specified syntactic, semantic and lexical constraints. We describe the details of the method and generation system, and then present results of experiments applying our method to probe for compositional information in embeddings from a number of existing sentence composition models. We find that the method is able to extract useful information about the differing capacities of these models, and we discuss the implications of our results with respect to these systems' capturing of sentence information. We make available for public use the datasets used for these experiments, as well as the generation system.