mlsanityData to 14 Oct 2026

A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities

arXiv 2609.03687 ↗First posted 3 Sep 2026cs.CLcited by 0

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

Coreference resolution is an important task in contextual reasoning. In this paper, we investigate the mechanism for representing and retrieving singular and plural entities for plural reference. We use a combination of mechanistic interpretability and attention pattern analysis to study the process in which LLMs predict a pronoun to refer back to previously mentioned entities. Using a range of causal intervention techniques, we find a set of attention heads that are responsible for (1) representing coreference information in the input, (2) identifying entities that form a plural reference, (3) transferring the information to the component that is responsible for selecting the antecedents and predicting the pronoun. We also find that LLMs align with humans in preference for plural pronoun. Specifically, entities in a plural construction are more likely to be referred to as a plural entity if they are ontologically similar and are linked by the conjunction "and".

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 29 of its 53 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

29 of 53 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  11. 2021Causal Abstractions of Neural Networkscited by 422
  12. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
17 more
  1. 2017End-to-end Neural Coreference Resolutioncited by 241
  2. 2023Localizing Model Behavior with Path Patchingcited by 195
  3. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  4. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  5. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  6. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  7. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
  8. 2022Coreference Resolution through a seq2seq Transition-Based Systemcited by 34
  9. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inferencecited by 29
  10. 2024Representational Analysis of Binding in Language Modelscited by 28
  11. 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 23
  12. 2024Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trendscited by 19
  13. 2023Are Large Language Models Robust Coreference Resolvers?cited by 15
  14. 2020Free the Plural: Unrestricted Split-Antecedent Anaphora Resolutioncited by 7
  15. 2026Cell-Based Representation of Relational Binding in Language Modelscited by 5
  16. 2022Scoring Coreference Chains with Split-Antecedent Anaphorscited by 4
  17. 2023Large Language Models Are Partially Primed in Pronoun Interpretationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
  2. 2026Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs
  3. 2026Beyond Hooking Onto the World: Referential Profiles and the Numerical Structure of LLM Grounding
  4. 2025How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs
  5. 2025Is analogy enough to draw novel adjective-noun inferences?
  6. 2024LLMs as Models for Analogical Reasoning
  7. 2024Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models
  8. 2025Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs