# A polar coordinate system represents syntax in large language models

arXiv: https://arxiv.org/abs/2412.05571 · HTML: https://mlsanity.com/paper/2412.05571/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 7 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Embedding Text in Hyperbolic Spaces](https://mlsanity.com/paper/1806.04313/) (2018)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Discovering the Compositional Structure of Vector Representations with Role Learning Networks](https://mlsanity.com/paper/1910.09113/) (2019)
- [A Non-Linear Structural Probe](https://mlsanity.com/paper/2105.10185/) (2021)
- [Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps](https://mlsanity.com/paper/2207.03380/) (2022)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [Introducing Orthogonal Constraint in Structural Probes](https://mlsanity.com/paper/2012.15228/) (2020)
- [Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context](https://mlsanity.com/paper/2302.14389/) (2023)
- [Can RNNs learn Recursive Nested Subject-Verb Agreements?](https://mlsanity.com/paper/2101.02258/) (2021)
- [Probing for Labeled Dependency Trees](https://mlsanity.com/paper/2203.12971/) (2022)

## Built on by (most cited first, 25 of 25)

- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [Deriving Neural Scaling Laws from the statistics of natural language](https://mlsanity.com/paper/2602.07488/) (2026)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Probing Syntax in Large Language Models: Successes and Remaining Challenges](https://mlsanity.com/paper/2508.03211/) (2025)
- [Deep networks learn to parse uniform-depth context-free languages from local statistics](https://mlsanity.com/paper/2602.06065/) (2026)
- [Sparse Reward Subsystem in Large Language Models](https://mlsanity.com/paper/2602.00986/) (2026)
- [Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures](https://mlsanity.com/paper/2509.25045/) (2025)
- [Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks](https://mlsanity.com/paper/2601.18617/) (2026)
- [Polar probe linearly decodes semantic structures from LLMs](https://mlsanity.com/paper/2605.14125/) (2026)
- [Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees](https://mlsanity.com/paper/2512.01870/) (2025)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Emergent Causal-Geometric Dynamics Across Depth in Large Language Models](https://mlsanity.com/paper/2602.04931/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Geometry-Guided Self-Supervision for Ultra-Fine-Grained Recognition with Limited Data](https://mlsanity.com/paper/2604.19345/) (2026)
- [Representational Curvature Modulates Behavioral Uncertainty in Large Language Models](https://mlsanity.com/paper/2604.23985/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames](https://mlsanity.com/paper/2605.29634/) (2026)
- [Binding Phoneme Identity to Position in Neural Models and the Human Brain](https://mlsanity.com/paper/2610.15843/) (2026)

## Abstract (arXiv)

Originally formalized with symbolic representations, syntactic trees may also be effectively represented in the activations of large language models (LLMs). Indeed, a 'Structural Probe' can find a subspace of neural activations, where syntactically related words are relatively close to one-another. However, this syntactic code remains incomplete: the distance between the Structural Probe word embeddings can represent the existence but not the type and direction of syntactic relations. Here, we hypothesize that syntactic relations are, in fact, coded by the relative direction between nearby embeddings. To test this hypothesis, we introduce a 'Polar Probe' trained to read syntactic relations from both the distance and the direction between word embeddings. Our approach reveals three main findings. First, our Polar Probe successfully recovers the type and direction of syntactic relations, and substantially outperforms the Structural Probe by nearly two folds. Second, we confirm that this polar coordinate system exists in a low-dimensional subspace of the intermediate layers of many LLMs and becomes increasingly precise in the latest frontier models. Third, we demonstrate with a new benchmark that similar syntactic relations are coded similarly across the nested levels of syntactic trees. Overall, this work shows that LLMs spontaneously learn a geometry of neural activations that explicitly represents the main symbolic structures of linguistic theory.