mlsanityData to 7 Oct 2026

ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation

arXiv 2203.09176 ↗First posted 17 Mar 2022cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, it ODE Transformer, which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT'14 English-German and English-French benchmarks) at a slight cost in inference efficiency.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 45 of its 74 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

45 of 74 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
33 more
  1. 2018Universal Transformerscited by 613
  2. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  3. 2017Stable Architectures for Deep Neural Networkscited by 340
  4. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  5. 2015Highway Networkscited by 257
  6. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  7. 2020Understanding the Difficulty of Training Transformerscited by 210
  8. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  9. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  10. 2019The Evolved Transformercited by 180
  11. 2018Bottom-Up Abstractive Summarizationcited by 177
  12. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  13. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  14. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  15. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  16. 2021Facebook AI WMT21 News Translation Task Submissioncited by 57
  17. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  18. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  19. 2016PolyNet: A Pursuit of Structural Diversity in Very Deep Networkscited by 38
  20. 2018Training Deeper Neural Machine Translation Models with Transparent Attentioncited by 37
  21. 2018A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correctioncited by 36
  22. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  23. 2021Momentum Residual Neural Networkscited by 36
  24. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  25. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
  26. 2020Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learningcited by 26
  27. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  28. 2020DeLighT: Deep and Light-weight Transformercited by 17
  29. 2019Depth Growing for Neural Machine Translationcited by 13
  30. 2020Multiscale Collaborative Deep Models for Neural Machine Translationcited by 11
  31. 2019Lipschitz Constrained Parameter Initialization for Deep Transformerscited by 9
  32. 2020N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equationscited by 9
  33. 2021The NiuTrans Machine Translation Systems for WMT21cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 3; 2025: 10; 2026: 5.

26 papers.

Sort:

By area:

  1. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  2. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  3. 2022Revisiting Grammatical Error Correction Evaluation and Beyondcited by 18
  4. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  5. 2022Learning Multiscale Transformer Models for Sequence Generationcited by 7
  6. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  7. 2026Deep Delta Learningcited by 7
  8. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  9. 2025HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMscited by 6
  10. 2023TranSFormer: Slow-Fast Transformer for Machine Translationcited by 4
  11. 2024Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learningcited by 4
  12. 2025Language Models Are Implicitly Continuouscited by 4
14 more
  1. 2024FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Modelscited by 3
  2. 2024Learning Adaptive Hydrodynamic Models Using Neural ODEs in Complex Conditionscited by 3
  3. 2025IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Methodcited by 3
  4. 2022EIT: Enhanced Interactive Transformercited by 1
  5. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  6. 2025Latent Flow Transformercited by 1
  7. 2026Revisiting the Shape Convention of Transformer Language Modelscited by 1
  8. 2025Federated Spectral Graph Transformers Meet Neural Ordinary Differential Equations for Non-IID Graphscited by 0
  9. 2025A multilevel approach to accelerate the training of Transformerscited by 0
  10. 2025TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classificationcited by 0
  11. 2025IIET: Efficient Numerical Transformer via Implicit Iterative Euler Methodcited by 0
  12. 2026Continuous-Depth Transformers with Learned Control Dynamicscited by 0
  13. 2026ODE-Based Transformer Decoders for Iterative Sign Language Translationcited by 0
  14. 2026Learning Reduced-Order Dynamics with Singularity via Latent-Augmented Neural Ordinary Differential Equationscited by 0