# Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere

arXiv: https://arxiv.org/abs/2607.24502 · HTML: https://mlsanity.com/paper/2607.24502/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [On the Role of Attention Masks and LayerNorm in Transformers](https://mlsanity.com/paper/2405.18781/) (2024)
- [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024)
- [Clustering in Causal Attention Masking](https://mlsanity.com/paper/2411.04990/) (2024)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Synchronization on circles and spheres with nonlinear interactions](https://mlsanity.com/paper/2405.18273/) (2024)
- [The Asymptotic Behavior of Attention in Transformers](https://mlsanity.com/paper/2412.02682/) (2024)
- [Normalization in Attention Dynamics](https://mlsanity.com/paper/2510.22026/) (2025)
- [Multistability of Self-Attention Dynamics in Transformers](https://mlsanity.com/paper/2511.11553/) (2025)
- [Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers](https://mlsanity.com/paper/2602.10959/) (2026)
- [Deconstructing Positional Information: From Attention Logits to Training Biases](https://mlsanity.com/paper/2505.13027/) (2025)
- [A Lifting method for analyzing distributed synchronization on the unit sphere](https://mlsanity.com/paper/1805.02528/) (2018)
- [Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding](https://mlsanity.com/paper/2512.03058/) (2025)
- [Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention](https://mlsanity.com/paper/2605.04279/) (2026)
- [RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways](https://mlsanity.com/paper/2606.11275/) (2026)
- [Kuramoto Attention: Synchronizing Self-Attention on the Torus](https://mlsanity.com/paper/2606.11585/) (2026)
- [Attention as Frustrated Synchronization](https://mlsanity.com/paper/2606.18694/) (2026)

## Abstract (arXiv)

Rotary position embeddings (RoPE) modify attention scores through position-dependent rotations, but their effect on normalized token dynamics is not captured by the vanilla spherical self-attention model. We study the continuous-time dynamics obtained when queries and keys are rotated while values remain on the unit sphere. The resulting attention kernel is reversible and admits a sharp uniform softmax floor, yet the natural RoPE interaction energy has derivatives of both signs within one fixed nontrivial system. Every consensus state remains an equilibrium, and its transverse linearization is a reversible Markov operator whose kernel depends on the consensus point through its energy across RoPE planes. On a resonant single-frequency ring we derive an exact Bessel-aliasing spectrum, including non-coprime frequencies and the correct fixed-ring large-β asymptotics. Globally, closed hemispheres are invariant, while pairwise non-obtuse configurations and strict open semicircles contract with explicit half-angle and single-point tail bounds. These regional estimates instantiate a kernel-generic positivity principle with the sharp RoPE softmax floor. RoPE also selects an explicit score-flattening twisted branch; the generic resonant family is non-hyperbolic and linearly unstable, whereas an odd antipodal family becomes a hyperbolic saddle after quotienting global rotation. In multiple dimensions, the local consensus gap can depend non-monotonically on the allocation of energy across frequency planes, so no universal ordering by frequency is valid. Independent matrix, finite-difference, and nonlinear-flow computations cross-check the theorem boundaries and the reported constants.