Do Attention Heads Compete or Cooperate during Counting?
On the map: Mechanistic interpretability → Circuits and mechanistic analysis
Abstract (arXiv)
We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting, which is a crucial deductive step in many algorithms. In particular, we investigate the collaboration/competition among the attention heads: we ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction. Our work presents evidence that on the semantics of the counting task, attention heads behave as a pseudo-ensemble, but their outputs need to be aggregated in a non-uniform manner in order to create an encoding that conforms to the syntax. Our source code will be available upon publication.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. First posted on arXiv 10 Feb 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Attend or Perish: Benchmarking Attention in Algorithmic Reasoning
- 2026Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
- 2026Counting to Four is still a Chore for VLMs
- 2026Architecture, Not Scale: Circuit Localization in Large Language Models
- 2025Cognitive Mirrors: Exploring the Diverse Functional Roles of Attention Heads in LLM Reasoning
- 2026Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
- 2024LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems
- 2026Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms