mlsanityData to 13 Oct 2026

Sinkformers: Transformers with Doubly Stochastic Attention

arXiv 2110.11773 ↗First posted 22 Oct 2021cs.LG stat.MLcited by 137

Not on the map. Why?

Abstract (arXiv)

Attention based models such as Transformers involve pairwise interactions between data points, modeled with a learnable attention matrix. Importantly, this attention matrix is normalized with the SoftMax operator, which makes it row-wise stochastic. In this paper, we propose instead to use Sinkhorn's algorithm to make attention matrices doubly stochastic. We call the resulting model a Sinkformer. We show that the row-wise stochastic attention matrices in classical Transformers get close to doubly stochastic matrices as the number of epochs increases, justifying the use of Sinkhorn normalization as an informative prior. On the theoretical side, we show that, unlike the SoftMax operation, this normalization makes it possible to understand the iterations of self-attention modules as a discretized gradient-flow for the Wasserstein metric. We also show in the infinite number of samples limit that, when rescaling both attention matrices and depth, Sinkformers operate a heat diffusion. On the experimental side, we show that Sinkformers enhance model accuracy in vision and natural language processing tasks. In particular, on 3D shapes classification, Sinkformers lead to a significant improvement.

Key facts. Cited by 137 papers in the map (data through 13 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 22 Oct 2021.

Builds on

40 of 64 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  7. 2018Neural Ordinary Differential Equationscited by 3,750
  8. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,295
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
  10. 2020Point Transformercited by 1,465
  11. 2016An overview of gradient descent optimization algorithmscited by 1,028
  12. 2021Scaling Vision Transformerscited by 998
28 more
  1. 2020PCT: Point cloud transformercited by 731
  2. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  3. 2019Deep Equilibrium Modelscited by 606
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 465
  5. 2019Differentiable Convex Optimization Layerscited by 461
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 413
  7. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  8. 2019Augmented Neural ODEscited by 396
  9. 2018Invertible Residual Networkscited by 364
  10. 2020Sparse Sinkhorn Attentioncited by 251
  11. 2017Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Networkcited by 208
  12. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  13. 2021Efficient and Modular Implicit Differentiationcited by 201
  14. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  15. 2020The Lipschitz Constant of Self-Attentioncited by 172
  16. 2016On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimizationcited by 111
  17. 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 103
  18. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  19. 2018SparseMAP: Differentiable Sparse Structured Inferencecited by 62
  20. 2018Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distancecited by 56
  21. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  22. 2021Momentum Residual Neural Networkscited by 36
  23. 2020Spectral convergence of diffusion maps: improved error bounds and an alternative normalisationcited by 25
  24. 2017Manifold learning with bi-stochastic kernelscited by 19
  25. 2020A Functional Perspective on Learning Symmetric Functions with Neural Networkscited by 16
  26. 2018Stochastic Deep Networkscited by 13
  27. 2018Stochastic Training of Residual Networks: a Differential Equation Viewpointcited by 12
  28. 2021On the Regularity of Attentioncited by 10

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 18; 2024: 31; 2025: 38; 2026: 42.

137 papers.

Sort:

By area:

  1. 2023A mathematical perspective on Transformerscited by 161
  2. 2023The emergence of clusters in self-attention dynamicscited by 135
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2023How Smooth Is Attention?cited by 44
  5. 2024Transformers are Universal In-context Learnerscited by 42
  6. 2022Sparsity-Constrained Optimal Transportcited by 32
  7. 2024Clustering in Causal Attention Maskingcited by 32
  8. 2024Measure-to-measure interpolation using Transformerscited by 28
  9. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  10. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  11. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
  12. 2025The Mean-Field Dynamics of Transformerscited by 26
38 more
  1. 2023Learning Gaussian Mixtures Using the Wasserstein-Fisher-Rao Gradient Flowcited by 25
  2. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  3. 2023Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approachcited by 24
  4. 2023Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Modelscited by 22
  5. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  6. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  7. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 17
  8. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  9. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  10. 2023Unlocking Slot Attention by Changing Optimal Transport Costscited by 16
  11. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  12. 2022Rethinking Initialization of the Sinkhorn Algorithmcited by 14
  13. 2023Unsupervised Manifold Linearizing and Clusteringcited by 12
  14. 2023SNEkhorn: Dimension Reduction with Symmetric Entropic Affinitiescited by 12
  15. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 12
  16. 2022Designing Robust Transformers using Robust Kernel Density Estimationcited by 11
  17. 2025Synchronization of mean-field models on the circlecited by 11
  18. 2023Non-aligned supervision for Real Image Dehazingcited by 9
  19. 2024OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentationcited by 9
  20. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  21. 2025Hierarchical Refinement: Optimal Transport to Infinity and Beyondcited by 9
  22. 2025Normalization in Attention Dynamicscited by 9
  23. 2026FlashSinkhorn: IO-Aware Entropic Optimal Transport on GPUcited by 9
  24. 2023Stochastic Latent Transformer: Efficient Modelling of Stochastically Forced Zonal Jetscited by 8
  25. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  26. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
  27. 2023Unbalanced Low-rank Optimal Transport Solverscited by 7
  28. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  29. 2024Elliptical Attentioncited by 7
  30. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  31. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  32. 2023Doubly Stochastic Graph-based Non-autoregressive Reaction Predictioncited by 6
  33. 2025Quantum Doubly Stochastic Transformerscited by 6
  34. 2025Transformers through the lens of support-preserving maps between measurescited by 6
  35. 2025Multistability of Self-Attention Dynamics in Transformerscited by 6
  36. 2026Clustering in Deep Stochastic Transformerscited by 6
  37. 2024Iterated Schr"odinger bridge approximation to Wasserstein Gradient Flowscited by 5
  38. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5

50 most cited of 137 All 137 (JSON)