# Relation Geometry in Semantic Space of Language Models

arXiv: https://arxiv.org/abs/2607.26762 · HTML: https://mlsanity.com/paper/2607.26762/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Knowledge Graph Embedding: A Survey from the Perspective of Representation Spaces](https://mlsanity.com/paper/2211.03536/) (2022)
- [Talking Heads: Understanding Inter-layer Communication in Transformer Language Models](https://mlsanity.com/paper/2406.09519/) (2024)
- [Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://mlsanity.com/paper/2211.07443/) (2022)
- [Introducing Orthogonal Constraint in Structural Probes](https://mlsanity.com/paper/2012.15228/) (2020)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [Meta-Learning a Cross-lingual Manifold for Semantic Parsing](https://mlsanity.com/paper/2209.12577/) (2022)
- [Do Large Language Models Understand Word Senses?](https://mlsanity.com/paper/2509.13905/) (2025)
- [Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy](https://mlsanity.com/paper/2106.13553/) (2021)
- [Analyzing Semantic Change through Lexical Replacements](https://mlsanity.com/paper/2404.18570/) (2024)
- [What relations are reliably embeddable in Euclidean space?](https://mlsanity.com/paper/1903.05347/) (2019)
- [Linguistic Blind Spots of Large Language Models](https://mlsanity.com/paper/2503.19260/) (2025)
- [Semantic Change Characterization with LLMs using Rhetorics](https://mlsanity.com/paper/2407.16624/) (2024)
- [A Non-commutative Bilinear Model for Answering Path Queries in Knowledge Graphs](https://mlsanity.com/paper/1909.01567/) (2019)
- [Rethinking Word Similarity: Semantic Similarity through Classification Confusion](https://mlsanity.com/paper/2502.05704/) (2025)
- [On the Distinctive Co-occurrence Characteristics of Antonymy](https://mlsanity.com/paper/2509.11534/) (2025)
- [In the LLM era, Word Sense Induction remains unsolved](https://mlsanity.com/paper/2603.11686/) (2026)
- [SpaceE: Knowledge Graph Embedding by Relational Linear Transformation in the Entity Space](https://mlsanity.com/paper/2204.10245/) (2022)
- [The quasi-semantic competence of LLMs: a case study on the part-whole relation](https://mlsanity.com/paper/2504.02395/) (2025)

## Built on by (most cited first, 1 of 1)

- [Evaluating Semantic Relation Knowledge through Bidirectional Relation Prediction from Gloss Pairs](https://mlsanity.com/paper/2610.14442/) (2026)

## Abstract (arXiv)

When it comes to generating vector representations of words, current language models are achieving high-quality results. However, what is not known is the extent to which knowledge about semantic relations is represented in the geometry of the semantic spaces created in this way. In order to answer this question, we study the relation geometry of such semantic spaces from three perspectives. We first examine whether words standing in a particular relation to a target word~(called relata) occupy the same region in semantic space, and whether the regions corresponding to different relations are distinct from each other. We then verify to what extent semantic spaces reflect certain well-known properties of relations, such as symmetry, asymmetry, and transitivity. Finally, we consider which information about the target words and relata is more important for relation geometry: their surface forms, or their contexts. We conduct experiments on six semantic relations using causal, masked, and diffusion language models. The results show that relata in asymmetric relations relatively clearly occupy a distinct region in semantic space. Asymmetric relations' properties are only moderately well encoded in the semantic space, yet better than those of symmetric ones. Furthermore, when considering the question which information source has the strongest impact on results amongst the models we evaluated, we find that lexical information tends to be more important for the causal language model, whereas contextual information is more important for the masked and diffusion language models. Our results empirically show that relation geometry is not equally well-represented for all relations in semantic space, suggesting that there is a difference in how well semantic relations might be learned from distributional information alone.