mlsanityData to 7 Oct 2026

ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation

arXiv 2104.02308 ↗First posted 6 Apr 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

It has been found that residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODEs). In this paper, we explore a deeper relationship between Transformer and numerical methods of ODEs. We show that a residual block of layers in Transformer can be described as a higher-order solution to ODEs. This leads us to design a new architecture (call it ODE Transformer) analogous to the Runge-Kutta method that is well motivated in ODEs. As a natural extension to Transformer, ODE Transformer is easy to implement and parameter efficient. Our experiments on three WMT tasks demonstrate the genericity of this model, and large improvements in performance over several strong baselines. It achieves 30.76 and 44.11 BLEU scores on the WMT'14 En-De and En-Fr test data. This sets a new state-of-the-art on the WMT'14 En-Fr task.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

30 of 43 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2018Universal Transformerscited by 613
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  10. 2017Stable Architectures for Deep Neural Networkscited by 340
  11. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  12. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
18 more
  1. 2020Understanding the Difficulty of Training Transformerscited by 210
  2. 2018Scaling Neural Machine Translationcited by 204
  3. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  4. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  5. 2019The Evolved Transformercited by 180
  6. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  7. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  8. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  9. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  10. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  11. 2016PolyNet: A Pursuit of Structural Diversity in Very Deep Networkscited by 38
  12. 2018Training Deeper Neural Machine Translation Models with Transparent Attentioncited by 37
  13. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  14. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  15. 2020DeLighT: Deep and Light-weight Transformercited by 17
  16. 2019Depth Growing for Neural Machine Translationcited by 13
  17. 2020Multiscale Collaborative Deep Models for Neural Machine Translationcited by 11
  18. 2019Lipschitz Constrained Parameter Initialization for Deep Transformerscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 1; 2023: 1; 2024: 2; 2025: 4; 2026: 2.

12 papers.

  1. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  2. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 12
  3. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  4. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  5. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  6. 2023Efficiency 360: Efficient Vision Transformerscited by 6
  7. 2021An Effective Non-Autoregressive Model for Spoken Language Understandingcited by 5
  8. 2021The NiuTrans Machine Translation Systems for WMT21cited by 4
  9. 2022Capture Salient Historical Information: A Fast and Accurate Non-Autoregressive Model for Multi-turn Spoken Language Understandingcited by 2
  10. 2026Layer-Parallel Training for Transformerscited by 1
  11. 2025Federated Spectral Graph Transformers Meet Neural Ordinary Differential Equations for Non-IID Graphscited by 0
  12. 2026Transformer as an Euler Discretization of Score-based Variational Flowcited by 0