Compositional Structures in Neural Embedding and Interaction Decompositions
Not on the map. Why?
Abstract (arXiv)
We describe a basic correspondence between linear algebraic structures within vector embeddings in artificial neural networks and conditional independence constraints on the probability distributions modeled by these networks. Our framework aims to shed light on the emergence of structural patterns in data representations, a phenomenon widely acknowledged but arguably still lacking a solid formal grounding. Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 12 Jul 2024.
Builds on
22 of 40 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
- 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
- 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
10 more
- 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
- 2023Language Models Represent Space and Timecited by 316
- 2018Towards a Definition of Disentangled Representationscited by 296
- 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
- 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
- 2024On the Origins of Linear Representations in Large Language Modelscited by 79
- 2019Analogies Explained: Towards Understanding Word Embeddingscited by 64
- 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
- 2021Audio Transformerscited by 41
- 2017Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Surveycited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.