# In-Context Binding Capacity in Language Models

arXiv: https://arxiv.org/abs/2609.30634 · HTML: https://mlsanity.com/paper/2609.30634/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 27 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Birth of a Transformer: A Memory Viewpoint](https://mlsanity.com/paper/2306.00802/) (2023)
- [Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws](https://mlsanity.com/paper/2404.05405/) (2024)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Statler: State-Maintaining Language Models for Embodied Reasoning](https://mlsanity.com/paper/2306.17840/) (2023)
- [Code Pretraining Improves Entity Tracking Abilities of Language Models](https://mlsanity.com/paper/2405.21068/) (2024)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context](https://mlsanity.com/paper/2510.06182/) (2025)
- [Working Memory Capacity of ChatGPT: An Empirical Study](https://mlsanity.com/paper/2305.03731/) (2023)
- [Unable to Forget: Proactive Interference Reveals Working Memory Limits in LLMs Beyond Context Length](https://mlsanity.com/paper/2506.08184/) (2025)
- [From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data](https://mlsanity.com/paper/2406.19292/) (2024)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models](https://mlsanity.com/paper/2606.05378/) (2026)
- [In-context superposition: human-like working memory interference in large language models](https://mlsanity.com/paper/2604.09670/) (2026)
- [Comparing Transformers and Hybrid Models at the Token Level](https://mlsanity.com/paper/2606.20936/) (2026)

## Abstract (arXiv)

How many assignments can a language model recall before it loses track of which value belongs to which entity? We measure this limit using continuous recall curves for 12 models at or below 3B parameters and a threshold sweep over 30 open models up to 12B. On the continuous curves, the load at which recall falls halfway to chance follows K50=cNα, with α=0.820 and R2=0.73. The broader sweep shows an eightfold range associated with pretraining recipe, although the continuous curves show no detectable recipe effect after controlling for scale, with few modern models in the fit. We derive why interference can lower measured capacity by reducing single-binding recall even when the load-dependent recall profile is unchanged. Direct task training also exceeds the extrapolated zero-shot law, but different measurement criteria prevent interpreting that comparison as a capacity gain. Its formation times follow a power-law form in two independent codebases, conditional on runs that succeed. Together, these results characterize capacity at the model's query interface. Bounds on joint recall and a decomposition of policy errors connect this measurement to working memory and instruction following, without treating recall as a measure of alignment. The controlled task also provides a baseline for testing whether binding limits constrain world-state tracking; the present experiments do not measure state updates or downstream transfer.