# A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization

arXiv: https://arxiv.org/abs/2506.06179 · HTML: https://mlsanity.com/paper/2506.06179/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2025.

## Built on by (most cited first, 5 of 5)

- [Internal Planning in Language Models: Characterizing Horizon and Branch Awareness](https://mlsanity.com/paper/2509.25260/) (2025)
- [Transformer-Based Multi-Agent Reinforcement Learning for Networked Systems with Long-Range Interactions](https://mlsanity.com/paper/2511.13103/) (2025)
- [Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers](https://mlsanity.com/paper/2602.02707/) (2026)
- [Analogies between Transformer Layers and Power Method](https://mlsanity.com/paper/2605.25619/) (2026)
- [Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers](https://mlsanity.com/paper/2610.13518/) (2026)

## Abstract (arXiv)

Self-attention has emerged as a core component of modern neural architectures, yet its theoretical underpinnings remain elusive. In this paper, we study self-attention through the lens of interacting entities, ranging from agents in multi-agent reinforcement learning to alleles in genetic sequences, and show that a single layer linear self-attention can efficiently represent, learn, and generalize functions capturing pairwise interactions, including out-of-distribution scenarios. Our analysis reveals that self-attention acts as a mutual interaction learner under minimal assumptions on the diversity of interaction patterns observed during training, thereby encompassing a wide variety of real-world domains. In addition, we validate our theoretical insights through experiments demonstrating that self-attention learns interaction functions and generalizes across both population distributions and out-of-distribution scenarios. Building on our theories, we introduce HyperFeatureAttention, a novel neural network module designed to learn couplings of different feature-level interactions between entities. Furthermore, we propose HyperAttention, a new module that extends beyond pairwise interactions to capture multi-entity dependencies, such as three-way, four-way, or general n-way interactions.