mlsanityData to 14 Oct 2026

How do Language Models Bind Entities in Context?

arXiv 2310.17191 ↗First posted 26 Oct 2023cs.LG cs.AI cs.CLcited by 108

Not on the map. Why?

Abstract (arXiv)

To correctly use in-context information, language models (LMs) must bind entities to their attributes. For example, given a context describing a "green square" and a "blue circle", LMs must bind the shapes to their respective colors. We analyze LM representations and identify the binding ID mechanism: a general mechanism for solving the binding problem, which we observe in every sufficiently large model from the Pythia and LLaMA families. Using causal interventions, we show that LMs' internal activations represent binding information by attaching binding ID vectors to corresponding entities and attributes. We further show that binding ID vectors form a continuous subspace, in which distances between binding ID vectors reflect their discernability. Overall, our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context, providing a step towards understanding general in-context reasoning in large-scale LMs.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

26 of 37 references.

References · 26

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2019Designing and Interpreting Probes with Control Taskscited by 640
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 601
14 more
  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  3. 2021Causal Abstractions of Neural Networkscited by 422
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  5. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  6. 2021Unsolved Problems in ML Safetycited by 292
  7. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  8. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  9. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  10. 2020Compositional Explanations of Neuronscited by 172
  11. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  12. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  13. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
  14. 2023Entity Tracking in Language Modelscited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 28; 2026: 55.

108 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  3. 2024How to use and interpret activation patchingcited by 214
  4. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  5. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  6. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  7. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  8. 2024Universal Neurons in GPT2 Language Modelscited by 102
  9. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 79
  10. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79
  11. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  12. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 46
38 more
  1. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 43
  2. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 41
  3. 2025Language Models use Lookbacks to Track Beliefscited by 39
  4. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  5. 2024Representational Analysis of Binding in Language Modelscited by 28
  6. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  7. 2025Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Contextcited by 26
  8. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 26
  9. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decompositioncited by 23
  10. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  11. 2025How Do LLMs Perform Two-Hop Reasoning in Context?cited by 20
  12. 2025Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?cited by 19
  13. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  14. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  15. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  16. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 14
  17. 2024Evidence from counterfactual tasks supports emergent analogical reasoning in large language modelscited by 14
  18. 2025Language Models' Factuality Depends on the Language of Inquirycited by 14
  19. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Modelscited by 14
  20. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objectscited by 13
  21. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  22. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language modelscited by 13
  23. 2024Characterizing the Role of Similarity in the Property Inferences of Language Modelscited by 11
  24. 2025How Causal Abstraction Underpins Computational Explanationcited by 11
  25. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  26. 2024Can Transformers Do Enumerative Geometry?cited by 8
  27. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  28. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 8
  29. 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 8
  30. 2025LLMs Process Lists With General Filter Headscited by 7
  31. 2025Addressing divergent representations from causal interventions on neural networkscited by 7
  32. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 6
  33. 2025Decoupling Positional and Symbolic Attention Behavior in Transformerscited by 6
  34. 2026Do Language Models Track Entities Across State Changes?cited by 6
  35. 2024Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neurosciencecited by 5
  36. 2025Investigating Mechanisms for In-Context Vision Language Bindingcited by 5
  37. 2025Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Surveycited by 5
  38. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5

50 most cited of 108 All 108 (JSON)