# Synchronization on circles and spheres with nonlinear interactions

arXiv: https://arxiv.org/abs/2405.18273 · HTML: https://mlsanity.com/paper/2405.18273/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 14 of 14)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity](https://mlsanity.com/paper/2106.15933/) (2021)
- [Saddle-to-Saddle Dynamics in Diagonal Linear Networks](https://mlsanity.com/paper/2304.00488/) (2023)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [Incremental Learning in Diagonal Linear Networks](https://mlsanity.com/paper/2208.14673/) (2022)
- [On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimization](https://mlsanity.com/paper/1809.11083/) (2018)
- [Expander graphs are globally synchronizing](https://mlsanity.com/paper/2210.12788/) (2022)
- [Benign landscapes of low-dimensional relaxations for orthogonal synchronization on general graphs](https://mlsanity.com/paper/2307.02941/) (2023)
- [Synchronization on Riemannian manifolds: Multiply connected implies multistable](https://mlsanity.com/paper/1906.07452/) (2019)

## Built on by (most cited first, 17 of 17)

- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [Clustering in Causal Attention Masking](https://mlsanity.com/paper/2411.04990/) (2024)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Measure-to-measure interpolation using Transformers](https://mlsanity.com/paper/2411.04551/) (2024)
- [Quantitative Clustering in Mean-Field Transformer Models](https://mlsanity.com/paper/2504.14697/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Synchronization of mean-field models on the circle](https://mlsanity.com/paper/2507.22857/) (2025)
- [Normalization in Attention Dynamics](https://mlsanity.com/paper/2510.22026/) (2025)
- [Perceptrons and localization of attention's mean-field landscape](https://mlsanity.com/paper/2601.21366/) (2026)
- [Clustering in Deep Stochastic Transformers](https://mlsanity.com/paper/2601.21942/) (2026)
- [On the number of modes of Gaussian kernel density estimators](https://mlsanity.com/paper/2412.09080/) (2024)
- [Random Quadratic Form on a Sphere: Synchronization by Common Noise](https://mlsanity.com/paper/2603.06187/) (2026)
- [YuriiFormer: A Suite of Nesterov-Accelerated Transformers](https://mlsanity.com/paper/2601.23236/) (2026)
- [Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention](https://mlsanity.com/paper/2605.04279/) (2026)
- [Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers](https://mlsanity.com/paper/2605.07772/) (2026)
- [Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows](https://mlsanity.com/paper/2605.18870/) (2026)
- [Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere](https://mlsanity.com/paper/2607.24502/) (2026)

## Abstract (arXiv)

We consider the dynamics of n points on a sphere in ℝd (d ≥ 2) which attract each other according to a function varphi of their inner products. When varphi is linear (varphi(t) = t), the points converge to a common value (i.e., synchronize) in various connectivity scenarios: this is part of classical work on Kuramoto oscillator networks. When varphi is exponential (varphi(t) = eβ t), these dynamics correspond to a limit of how idealized transformers process data, as described by Geshkovski et al. (2025). Accordingly, they ask whether synchronization occurs for exponential varphi. The answer depends on the dimension d. In the context of consensus for multi-agent control, Markdahl et al. (2018) show that for d ≥ 3 (spheres), if the interaction graph is connected and varphi is increasing and convex, then the system synchronizes. We give a separate proof of this result. What is the situation on circles (d=2)? First, we show that varphi being increasing and convex is no longer sufficient (even for complete graphs). Then we identify a new condition under which we do have synchronization on the circle (namely, if the Taylor coefficients of varphi' are decreasing). As a corollary, this provide synchronization for exponential varphi with β ∈ (0, 1]. The proofs are based on nonconvex landscape analysis.