# Polar probe linearly decodes semantic structures from LLMs

arXiv: https://arxiv.org/abs/2605.14125 · HTML: https://mlsanity.com/paper/2605.14125/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [ICLR: In-Context Learning of Representations](https://mlsanity.com/paper/2501.00070/) (2024)
- [Language Models Encode Numbers Using Digit Representations in Base 10](https://mlsanity.com/paper/2410.11781/) (2024)
- [Linear Representations of Political Perspective Emerge in Large Language Models](https://mlsanity.com/paper/2503.02080/) (2025)
- [Representational Analysis of Binding in Language Models](https://mlsanity.com/paper/2409.05448/) (2024)
- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [Introducing Orthogonal Constraint in Structural Probes](https://mlsanity.com/paper/2012.15228/) (2020)
- [Probing for Labeled Dependency Trees](https://mlsanity.com/paper/2203.12971/) (2022)
- [More than Correlation: Do Large Language Models Learn Causal Representations of Space?](https://mlsanity.com/paper/2312.16257/) (2023)
- [Probing Syntax in Large Language Models: Successes and Remaining Challenges](https://mlsanity.com/paper/2508.03211/) (2025)

## Built on by (most cited first, 2 of 2)

- [Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams](https://mlsanity.com/paper/2605.25310/) (2026)
- [Binding Phoneme Identity to Position in Neural Models and the Human Brain](https://mlsanity.com/paper/2610.15843/) (2026)

## Abstract (arXiv)

How do artificial neural networks bind concepts to form complex semantic structures? Here, we propose a simple neural code, whereby the existence and the type of relations between entities are represented by the distance and the direction between their embeddings, respectively. We test this hypothesis in a variety of Large Language Models (LLMs), each input with natural-language descriptions of minimalist tasks from five different domains: arithmetic, visual scenes, family trees, metro maps and social interactions. Results show that the true semantic structures can be linearly recovered with a Polar Probe targeting a subspace of LLMs' layer activations. Second, this code emerges mostly in middle layers and improves with LLM performance. Third, these Polar Probes successfully generalize to new entities and relation types, but degrades with the size of the semantic structure. Finally, the quality of the polar representation correlates with the LLM's ability to answer questions about the semantic structure. Together, these findings suggest that LLMs learn to build complex semantic structures by binding representations with a simple geometrical principle.