Synchronization on circles and spheres with nonlinear interactions
Not on the map. Why?
Abstract (arXiv)
We consider the dynamics of n points on a sphere in ℝd (d ≥ 2) which attract each other according to a function varphi of their inner products. When varphi is linear (varphi(t) = t), the points converge to a common value (i.e., synchronize) in various connectivity scenarios: this is part of classical work on Kuramoto oscillator networks. When varphi is exponential (varphi(t) = eβ t), these dynamics correspond to a limit of how idealized transformers process data, as described by Geshkovski et al. (2025). Accordingly, they ask whether synchronization occurs for exponential varphi. The answer depends on the dimension d. In the context of consensus for multi-agent control, Markdahl et al. (2018) show that for d ≥ 3 (spheres), if the interaction graph is connected and varphi is increasing and convex, then the system synchronizes. We give a separate proof of this result. What is the situation on circles (d=2)? First, we show that varphi being increasing and convex is no longer sufficient (even for complete graphs). Then we identify a new condition under which we do have synchronization on the circle (namely, if the Taylor coefficients of varphi' are decreasing). As a corollary, this provide synchronization for exponential varphi with β ∈ (0, 1]. The proofs are based on nonconvex landscape analysis.
Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 14 of its 57 references that are in the map. First posted on arXiv 28 May 2024.
Builds on
14 of 57 references.
References · 14
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2017Stable Architectures for Deep Neural Networkscited by 341
- 2023A mathematical perspective on Transformerscited by 164
- 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
- 2013Manopt, a Matlab toolbox for optimization on manifoldscited by 113
- 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
- 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
- 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
- 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
- 2022Incremental Learning in Diagonal Linear Networkscited by 28
- 2018On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimizationcited by 11
- 2022Expander graphs are globally synchronizingcited by 8
Built on by
Citing papers per year: 2024: 4; 2025: 5; 2026: 8.
17 papers.
Sort:
- 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
- 2024Clustering in Causal Attention Maskingcited by 32
- 2025The Mean-Field Dynamics of Transformerscited by 30
- 2024Measure-to-measure interpolation using Transformerscited by 29
- 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
- 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
- 2025Synchronization of mean-field models on the circlecited by 12
- 2025Normalization in Attention Dynamicscited by 9
- 2026Perceptrons and localization of attention's mean-field landscapecited by 8
- 2026Clustering in Deep Stochastic Transformerscited by 8
- 2024On the number of modes of Gaussian kernel density estimatorscited by 4
- 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 4
5 more
- 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformerscited by 2
- 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 2
- 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
- 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 1
- 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0
- 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere27 Jul 2026
- 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows15 May 2026
- 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers8 May 2026
- 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention5 May 2026
- 2026Random Quadratic Form on a Sphere: Synchronization by Common Noise6 Mar 2026
- 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformers30 Jan 2026
- 2026Clustering in Deep Stochastic Transformers29 Jan 2026
- 2026Perceptrons and localization of attention's mean-field landscape29 Jan 2026
- 2025The Mean-Field Dynamics of Transformers1 Dec 2025
- 2025Normalization in Attention Dynamics24 Oct 2025
- 2025A multiscale analysis of mean-field transformers in the moderate interaction regime29 Sep 2025
- 2025Synchronization of mean-field models on the circle30 Jul 2025
5 more
- 2025Quantitative Clustering in Mean-Field Transformer Models20 Apr 2025
- 2024On the number of modes of Gaussian kernel density estimators12 Dec 2024
- 2024Clustering in Causal Attention Masking7 Nov 2024
- 2024Measure-to-measure interpolation using Transformers7 Nov 2024
- 2024Emergence of meta-stable clustering in mean-field transformer models30 Oct 2024
- 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
- 2024Clustering in Causal Attention Maskingcited by 32
- 2025The Mean-Field Dynamics of Transformerscited by 30
- 2024Measure-to-measure interpolation using Transformerscited by 29
- 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
- 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
- 2025Synchronization of mean-field models on the circlecited by 12
- 2025Normalization in Attention Dynamicscited by 9
- 2026Perceptrons and localization of attention's mean-field landscapecited by 8
- 2026Clustering in Deep Stochastic Transformerscited by 8
- 2024On the number of modes of Gaussian kernel density estimatorscited by 4
- 2026Random Quadratic Form on a Sphere: Synchronization by Common Noisecited by 4
12 most cited of 17 All 17 (JSON)
- 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere27 Jul 2026
- 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows15 May 2026
- 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers8 May 2026
- 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention5 May 2026
- 2026Random Quadratic Form on a Sphere: Synchronization by Common Noise6 Mar 2026
- 2026YuriiFormer: A Suite of Nesterov-Accelerated Transformers30 Jan 2026
- 2026Clustering in Deep Stochastic Transformers29 Jan 2026
- 2026Perceptrons and localization of attention's mean-field landscape29 Jan 2026
- 2025The Mean-Field Dynamics of Transformers1 Dec 2025
- 2025Normalization in Attention Dynamics24 Oct 2025
- 2025A multiscale analysis of mean-field transformers in the moderate interaction regime29 Sep 2025
- 2025Synchronization of mean-field models on the circle30 Jul 2025
12 newest of 17 All 17 (JSON)