mlsanityData to 14 Oct 2026

OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization

arXiv 2501.18793 ↗First posted 30 Jan 2025cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by transformer blocks. We leverage optimal transport theory to regularize the training problem, which enhances stability in training and improves generalization of the resulting model. Moreover, we demonstrate in theory that this regularization is necessary as it promotes uniqueness and regularity of solutions. Our model is flexible in that almost any existing transformer architectures can be adopted to construct the dynamical system with only slight modifications to the existing code. We perform extensive numerical experiments on tasks motivated by natural language processing, image classification, and point cloud classification. Our experimental results show that the proposed method improves the performance of its discrete counterpart and outperforms relevant comparing models.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

37 of 61 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2018Neural Ordinary Differential Equationscited by 3,779
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2020On Layer Normalization in the Transformer Architecturecited by 922
  9. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  10. 2017Stable Architectures for Deep Neural Networkscited by 341
  11. 2022Transformer for Partial Differential Equations' Operator Learningcited by 283
  12. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
25 more
  1. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  2. 2023A mathematical perspective on Transformerscited by 164
  3. 2020OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transportcited by 143
  4. 2023The emergence of clusters in self-attention dynamicscited by 140
  5. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  6. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  7. 2023Scaling transformer neural networks for skillful and reliable medium-range weather forecastingcited by 110
  8. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 101
  9. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  10. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 66
  11. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  12. 2020Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flowscited by 37
  13. 2024Clustering in Causal Attention Maskingcited by 32
  14. 2019Potential Flow Generator with L2 Optimal Transport Regularity for Generative Modelscited by 31
  15. 2024Measure-to-measure interpolation using Transformerscited by 29
  16. 2023A mean-field games laboratory for generative modelingcited by 28
  17. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 27
  18. 2022Bridging Mean-Field Games and Normalizing Flows with Trajectory Regularizationcited by 17
  19. 2022Taming Hyperparameter Tuning in Continuous Normalizing Flows Using the JKO Schemecited by 17
  20. 2022A Neural ODE Interpretation of Transformer Layerscited by 14
  21. 2021ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translationcited by 12
  22. 2020N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equationscited by 9
  23. 2024Universal Approximation of Mean-Field Models via Transformerscited by 7
  24. 2024A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODEcited by 4
  25. 2024Combining Wasserstein-1 and Wasserstein-2 proximals: robust manifold learning via well-posed generative flowscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026Deep Delta Learningcited by 9
  2. 2025Neuronal Attention Circuit (NAC) for Representation Learningcited by 3
  3. 2026FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learningcited by 1
  4. 2025Embedding Empirical Distributions for Computing Optimal Transport Mapscited by 0
  5. 2026From Self-Attention to Connection Laplacian: A Unified Operator View of Transformerscited by 0
  6. 2026Measuring Optimal Transport in Transformer Depthcited by 0
  7. 2026Continuous-Time Machine Learning: A Unified Mathematical Perspectivecited by 0