# Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context

arXiv: https://arxiv.org/abs/2510.06182 · HTML: https://mlsanity.com/paper/2510.06182/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Monitoring Latent World States in Language Models with Propositional Probes](https://mlsanity.com/paper/2406.19501/) (2024)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [Discovering Variable Binding Circuitry with Desiderata](https://mlsanity.com/paper/2307.03637/) (2023)
- [Representational Analysis of Binding in Language Models](https://mlsanity.com/paper/2409.05448/) (2024)
- [How Do Transformers Learn Variable Binding in Symbolic Programs?](https://mlsanity.com/paper/2505.20896/) (2025)
- [Aspects of human memory and Large Language Models](https://mlsanity.com/paper/2311.03839/) (2023)
- [How Causal Abstraction Underpins Computational Explanation](https://mlsanity.com/paper/2508.11214/) (2025)
- [Combining Causal Models for More Accurate Abstractions of Neural Networks](https://mlsanity.com/paper/2503.11429/) (2025)

## Built on by (most cited first, 26 of 26)

- [Do Language Models Track Entities Across State Changes?](https://mlsanity.com/paper/2605.30233/) (2026)
- [Cell-Based Representation of Relational Binding in Language Models](https://mlsanity.com/paper/2604.19052/) (2026)
- [Seeing to Generalize: How Visual Data Corrects Binding Shortcuts](https://mlsanity.com/paper/2602.15183/) (2026)
- [The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models](https://mlsanity.com/paper/2603.22278/) (2026)
- [Decomposing Query-Key Feature Interactions Using Contrastive Covariances](https://mlsanity.com/paper/2602.04752/) (2026)
- [A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models](https://mlsanity.com/paper/2606.08644/) (2026)
- [Data-driven Circuit Discovery for Interpretability of Language Models](https://mlsanity.com/paper/2605.09129/) (2026)
- [Slot Machines: How LLMs Keep Track of Multiple Entities](https://mlsanity.com/paper/2604.21139/) (2026)
- [Talk is Cheap, Communication is Hard: Dynamic Grounding Failures and Repair in Multi-Agent Negotiation](https://mlsanity.com/paper/2605.01750/) (2026)
- [Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstraction](https://mlsanity.com/paper/2605.02234/) (2026)
- [Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization](https://mlsanity.com/paper/2605.31558/) (2026)
- [Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages](https://mlsanity.com/paper/2608.04183/) (2026)
- [Legible Failures: Detecting and Repairing In-Context Binding Errors](https://mlsanity.com/paper/2609.11216/) (2026)
- [Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs](https://mlsanity.com/paper/2609.31193/) (2026)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)
- [Source-Modality Monitoring in Vision-Language Models](https://mlsanity.com/paper/2604.22038/) (2026)
- [A Mechanistic Understanding of Pronoun Fidelity in LLMs](https://mlsanity.com/paper/2606.16407/) (2026)
- [Vision-Language Models are Fragile Multilingual Associators](https://mlsanity.com/paper/2608.12333/) (2026)
- [The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space](https://mlsanity.com/paper/2609.00515/) (2026)
- [In-Context Binding Capacity in Language Models](https://mlsanity.com/paper/2609.30634/) (2026)
- [The Key Handoff: Retrieval in Hybrid Language Models](https://mlsanity.com/paper/2609.32942/) (2026)
- [Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context](https://mlsanity.com/paper/2609.35663/) (2026)
- [Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval](https://mlsanity.com/paper/2609.38530/) (2026)
- [When Context Changes: Understanding Update Failures in LLMs](https://mlsanity.com/paper/2609.38866/) (2026)
- [The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention](https://mlsanity.com/paper/2610.00910/) (2026)
- [When Goals Leak Across Relationships: How Cached Goal Representations Shape Cross-Client Decisions](https://mlsanity.com/paper/2610.17365/) (2026)

## Abstract (arXiv)

A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval. For example, an LM might represent "Ann loves pie" by binding "Ann" to "pie", allowing it to later retrieve "Ann" when asked "Who loves pie?" Prior research on short lists of bound entities found strong evidence that LMs implement such retrieval via a positional mechanism, where "Ann" is retrieved based on its position in context. In this work, we find that this mechanism generalizes poorly to more complex settings; as the number of bound entities in context increases, the positional mechanism becomes noisy and unreliable in middle positions. To compensate for this, we find that LMs supplement the positional mechanism with a lexical mechanism (retrieving "Ann" using its bound counterpart "pie") and a reflexive mechanism (retrieving "Ann" through a direct pointer). Through extensive experiments on nine models and ten binding tasks, we uncover a consistent pattern in how LMs mix these mechanisms to drive model behavior. We leverage these insights to develop a causal model combining all three mechanisms that estimates next token distributions with 95% agreement. Finally, we show that our model generalizes to substantially longer inputs of open-ended text interleaved with entity groups, further demonstrating the robustness of our findings in more natural settings. Overall, our study establishes a more complete picture of how LMs bind and retrieve entities in-context.