YuriiFormer: A Suite of Nesterov-Accelerated Transformers
Not on the map. Why?
Abstract (arXiv)
We propose a variational framework that interprets transformer layers as iterations of an optimization algorithm acting on token embeddings. In this view, self-attention implements a gradient step of an interaction energy, while MLP layers correspond to gradient updates of a potential energy. Standard GPT-style transformers emerge as vanilla gradient descent on the resulting composite objective, implemented via Lie--Trotter splitting between these two energy functionals. This perspective enables principled architectural design using classical optimization ideas. As a proof of concept, we introduce a Nesterov-style accelerated transformer that preserves the same attention and MLP oracles. The resulting architecture consistently outperforms a nanoGPT baseline on TinyStories and OpenWebText, demonstrating that optimization-theoretic insights can translate into practical gains.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
23 of 51 references.
References · 23
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
- 2017Stable Architectures for Deep Neural Networkscited by 341
- 2023A mathematical perspective on Transformerscited by 164
- 2023The emergence of clusters in self-attention dynamicscited by 140
- 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
- 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
- 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
- 2024Dynamic metastability in the self-attention modelcited by 46
- 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
11 more
- 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
- 2025The Mean-Field Dynamics of Transformerscited by 30
- 2022Transformers from an Optimization Perspectivecited by 27
- 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
- 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
- 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
- 2025Critical attention scaling in long-context transformerscited by 20
- 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
- 2025Synchronization of mean-field models on the circlecited by 12
- 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
- 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.