Attention as Binding: A Vector-Symbolic Perspective on Transformer Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Transformer-based language models display impressive reasoning-like behavior, yet remain brittle on tasks that require stable symbolic manipulation. This paper develops a unified perspective on these phenomena by interpreting self-attention and residual streams as implementing an approximate Vector Symbolic Architecture (VSA). In this view, queries and keys define role spaces, values encode fillers, attention weights perform soft unbinding, and residual connections realize superposition of many bound structures. We use this algebraic lens to relate transformer internals to chain-of-thought traces, program-based reasoning, and memory-augmented tool use, and to explain characteristic failure modes such as variable confusion and inconsistency across logically related prompts. Building on this perspective, we propose VSA-inspired architectural biases, including explicit binding/unbinding heads and hyperdimensional memory layers, and training objectives that promote role-filler separation and robust superposition. Finally, we outline metrics for measuring "VSA-likeness" and logical compositionality, and pose theoretical and architectural open problems. Overall, the paper argues that viewing attention as soft vector-symbolic computation offers a principled route toward more interpretable and logically reliable reasoning systems.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 78 references that are in the map. First posted on arXiv 8 Dec 2025.
Builds on
19 of 78 references.
References · 19
- 2023GPT-4 Technical Reportcited by 18,132
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2016Layer Normalizationcited by 5,579
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2022In-context Learning and Induction Headscited by 930
- 2014Neural Turing Machinescited by 630
7 more
- 2020Generative Language Modeling for Automated Theorem Provingcited by 379
- 2014Memory Networkscited by 354
- 2019Compositionality decomposed: how do neural networks generalise?cited by 264
- 2025Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completioncited by 22
- 2017Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networkscited by 18
- 2018DeepProbLog: Neural Probabilistic Logic Programmingcited by 5
- 2016Logic Tensor Networks: Deep Learning and Logical Reasoning from Data and Knowledgecited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
- 2025From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits
- 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
- 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
- 2026The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning
- 2025Attend or Perish: Benchmarking Attention in Algorithmic Reasoning
- 2026Unveiling the Reasoning Process of Large Language Models
- 2026Towards a Mechanistic Understanding of Propositional Logical Reasoning in Large Language Models