# Native Hierarchical and Compositional Representations with Subspace Embeddings

arXiv: https://arxiv.org/abs/2508.16687 · HTML: https://mlsanity.com/paper/2508.16687/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Representation Tradeoffs for Hyperbolic Embeddings](https://mlsanity.com/paper/1804.03329/) (2018)
- [Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings](https://mlsanity.com/paper/2002.05969/) (2020)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs](https://mlsanity.com/paper/2010.11465/) (2020)
- [Order-Embeddings of Images and Language](https://mlsanity.com/paper/1511.06361/) (2015)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs](https://mlsanity.com/paper/2110.13715/) (2021)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [VQA-LOL: Visual Question Answering under the Lens of Logic](https://mlsanity.com/paper/2002.08325/) (2020)
- [Hierarchical Image Classification using Entailment Cone Embeddings](https://mlsanity.com/paper/2004.03459/) (2020)
- [Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations](https://mlsanity.com/paper/2403.20312/) (2024)
- [Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones](https://mlsanity.com/paper/2110.14923/) (2021)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [Language Models as Hierarchy Encoders](https://mlsanity.com/paper/2401.11374/) (2024)
- [Hyperbolic Disk Embeddings for Directed Acyclic Graphs](https://mlsanity.com/paper/1902.04335/) (2019)
- [How and where does CLIP process negation?](https://mlsanity.com/paper/2407.10488/) (2024)
- [Hierarchical Density Order Embeddings](https://mlsanity.com/paper/1804.09843/) (2018)

## Abstract (arXiv)

Traditional embeddings represent datapoints as vectors, which makes similarity easy to compute but limits how well they capture hierarchies and compositionality. We propose a fundamentally different approach: representing concepts as linear subspaces. By spanning multiple dimensions, subspaces can model broader concepts with higher-dimensional regions and nest more specific concepts within them. This geometry naturally captures generality through dimension, hierarchy through inclusion, and enables an emergent structure for composition via linear algebraic operations. To make this paradigm trainable, we introduce a differentiable subspace parameterization via soft projection matrices, allowing the effective dimension of each subspace to be learned. Our method not only achieves state-of-the-art performance on hierarchical and natural language inference benchmarks but also provides a geometrically-grounded model of entailment. Further, we demonstrate that while standard vector embeddings degrade to near-random performance on negated queries, subspace embeddings natively capture logical composition without explicit supervision, while preserving compatibility with efficient Euclidean vector search.