# Representational Analysis of Binding in Language Models

arXiv: https://arxiv.org/abs/2409.05448 · HTML: https://mlsanity.com/paper/2409.05448/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 22 of 22)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Code Pretraining Improves Entity Tracking Abilities of Language Models](https://mlsanity.com/paper/2405.21068/) (2024)
- [A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations](https://mlsanity.com/paper/2109.04727/) (2021)
- [Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers](https://mlsanity.com/paper/2301.06758/) (2023)

## Built on by (most cited first, 28 of 28)

- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context](https://mlsanity.com/paper/2510.06182/) (2025)
- [How Do Transformers Learn Variable Binding in Symbolic Programs?](https://mlsanity.com/paper/2505.20896/) (2025)
- [How Causal Abstraction Underpins Computational Explanation](https://mlsanity.com/paper/2508.11214/) (2025)
- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [Addressing divergent representations from causal interventions on neural networks](https://mlsanity.com/paper/2511.04638/) (2025)
- [The Mechanistic Emergence of Symbol Grounding in Language Models](https://mlsanity.com/paper/2510.13796/) (2025)
- [Do Language Models Track Entities Across State Changes?](https://mlsanity.com/paper/2605.30233/) (2026)
- [In-Context Algebra](https://mlsanity.com/paper/2512.16902/) (2025)
- [Uncovering Grounding IDs: How External Cues Shape Multimodal Binding](https://mlsanity.com/paper/2509.24072/) (2025)
- [Seeing to Generalize: How Visual Data Corrects Binding Shortcuts](https://mlsanity.com/paper/2602.15183/) (2026)
- [The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models](https://mlsanity.com/paper/2603.22278/) (2026)
- [Decomposing Query-Key Feature Interactions Using Contrastive Covariances](https://mlsanity.com/paper/2602.04752/) (2026)
- [AdaSplash-2: Faster Differentiable Sparse Attention](https://mlsanity.com/paper/2604.15180/) (2026)
- [Polar probe linearly decodes semantic structures from LLMs](https://mlsanity.com/paper/2605.14125/) (2026)
- [Slot Machines: How LLMs Keep Track of Multiple Entities](https://mlsanity.com/paper/2604.21139/) (2026)
- [Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs](https://mlsanity.com/paper/2609.31193/) (2026)
- [When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question-Answering](https://mlsanity.com/paper/2601.03914/) (2026)
- [One mechanism for many mental spaces: a shared router over a value slot in language models](https://mlsanity.com/paper/2607.10248/) (2026)
- [Belief-reality separation lives in routing over a shared value slot in language models](https://mlsanity.com/paper/2607.11945/) (2026)
- [Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models](https://mlsanity.com/paper/2607.22575/) (2026)
- [Vision-Language Models are Fragile Multilingual Associators](https://mlsanity.com/paper/2608.12333/) (2026)
- [Syntax vs. Semantics: How Transformers Learn Deep Dependencies](https://mlsanity.com/paper/2608.26139/) (2026)
- [How Do Language Models Choose Between Context and Memory?](https://mlsanity.com/paper/2609.00753/) (2026)
- [A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities](https://mlsanity.com/paper/2609.03687/) (2026)
- [The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention](https://mlsanity.com/paper/2610.00910/) (2026)
- [Phantom Merge: When Your Large Language Model Agents Pick One but Tell You About Another](https://mlsanity.com/paper/2610.16141/) (2026)

## Abstract (arXiv)

Entity tracking is essential for complex reasoning. To perform in-context entity tracking, language models (LMs) must bind an entity to its attribute (e.g., bind a container to its content) to recall attribute for a given entity. For example, given a context mentioning ``The coffee is in Box Z, the stone is in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later, LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs, existing research introduces a Binding ID mechanism and states that LMs use a abstract concept called Binding ID (BI) to internally mark entity-attribute pairs. However, they have not captured the Ordering ID (OI) from entity activations that directly determines the binding behaviour. In this work, we provide a novel view of the BI mechanism by localizing OI and proving the causality between OI and binding behaviour. Specifically, by leveraging dimension reduction methods (e.g., PCA), we discover that there exists a low-rank subspace in the activations of LMs, that primarily encodes the order (i.e., OI) of entity and attribute. Moreover, we also discover the causal effect of OI on binding that when editing representations along the OI encoding direction, LMs tend to bind a given entity to other attributes accordingly. For example, by patching activations along the OI encoding direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z contains the map''.