mlsanityData to 7 Oct 2026

Attention as Frustrated Synchronization

arXiv 2606.18694 ↗First posted 17 Jun 2026cs.LG cond-mat.dis-nn cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

A network of oscillators that synchronizes perfectly computes nothing further, so an attention architecture built from synchronization must locate its computation in structured departures from agreement. We introduce the Frustrated Synchronization Network (FSN), whose token states are phases on a torus and whose entire value pathway is one learned complex coupling kernel over harmonics and a one-step delay. Each component of the kernel is a frustration in the sense of the synchronization literature. The complex phases are static Kuramoto-Sakaguchi frustration angles, the signed harmonics are repulsive Daido components, and the delay term, which couples each token to the successors of the tokens it attends to, is algebraically identical to Kuramoto-Sakaguchi coupling whose frustration angle is the data's own transition, so next-token prediction is implemented as synchronization frustrated by the data. At matched one-million-parameter and training budgets on character-level text and code, the FSN's validation loss is below a tuned RoPE-SwiGLU transformer's at every epoch measured, and the comparison survives training the baseline to convergence: every thirty-epoch enwik8 seed finishes below the transformer's converged fifty-epoch loss of 1.611, and the FSN's completed fifty-epoch runs converge to 1.5953 +/- 0.0014. A variant with every feed-forward block replaced by mean-field coupling to learned collective modes, leaving no multilayer perceptron in the stack, tracks the transformer. On natural text the unfrustrated base layer falls behind the converged transformer at every copy depth, worst on long-range copy events; the kernel reverses the deficit at every depth of four and beyond. Headline comparisons are at the one-million-parameter scale; a scale ladder is complete through four million parameters with the advantage persisting, and remaining arms are marked as in progress.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

13 of 21 references.

References · 13

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  5. 2020Hopfield Networks is All You Needcited by 461
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  7. 2022Recurrent Memory Transformercited by 197
  8. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  9. 2023A mathematical perspective on Transformerscited by 160
  10. 2023The emergence of clusters in self-attention dynamicscited by 133
  11. 2021What are higher-order networks?cited by 66
  12. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
1 more
  1. 2024Clustering in Causal Attention Maskingcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.