# Cell-Based Representation of Relational Binding in Language Models

arXiv: https://arxiv.org/abs/2604.19052 · HTML: https://mlsanity.com/paper/2604.19052/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Monitoring Latent World States in Language Models with Propositional Probes](https://mlsanity.com/paper/2406.19501/) (2024)
- [Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution](https://mlsanity.com/paper/2410.00153/) (2024)
- [Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context](https://mlsanity.com/paper/2510.06182/) (2025)
- [Large Language Models Encode Semantics and Alignment in Linearly Separable Representations](https://mlsanity.com/paper/2507.09709/) (2025)
- [Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial States](https://mlsanity.com/paper/2503.09066/) (2025)
- [Linear Spatial World Models Emerge in Large Language Models](https://mlsanity.com/paper/2506.02996/) (2025)

## Built on by (most cited first, 5 of 5)

- [Relational Rank Geometry in Transformers: Detecting and Steering Hidden-State Relation Frames](https://mlsanity.com/paper/2605.29634/) (2026)
- [How Do Language Models Choose Between Context and Memory?](https://mlsanity.com/paper/2609.00753/) (2026)
- [A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities](https://mlsanity.com/paper/2609.03687/) (2026)
- [Causal and Interpretable Structures in LLM Compositional Tasks](https://mlsanity.com/paper/2609.35970/) (2026)
- [The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention](https://mlsanity.com/paper/2610.00910/) (2026)

## Abstract (arXiv)

Understanding a discourse requires tracking entities and the relations that hold between them. While Large Language Models (LLMs) perform well on relational reasoning, the mechanism by which they bind entities, relations, and attributes remains unclear. We study discourse-level relational binding and show that LLMs encode it via a Cell-based Binding Representation (CBR): a low-dimensional linear subspace in which each ``cell'' corresponds to an entity--relation index pair, and bound attributes are retrieved from the corresponding cell during inference. Using controlled multi-sentence data annotated with entity and relation indices, we identify the CBR subspace by decoding these indices from attribute-token activations with Partial Least Squares regression. Across domains and two model families, the indices are linearly decodable and form a grid-like geometry in the projected space. We further find that context-specific CBR representations are related by translation vectors in activation space, enabling cross-context transfer. Finally, activation patching shows that manipulating this subspace systematically changes relational predictions and that perturbing it disrupts performance, providing causal evidence that LLMs rely on CBR for relational binding.