mlsanityData to 7 Oct 2026

A Neural ODE Interpretation of Transformer Layers

arXiv 2212.06011 ↗First posted 12 Dec 2022cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformer layers, which use an alternating pattern of multi-head attention and multi-layer perceptron (MLP) layers, provide an effective tool for a variety of machine learning problems. As the transformer layers use residual connections to avoid the problem of vanishing gradients, they can be viewed as the numerical integration of a differential equation. In this extended abstract, we build upon this connection and propose a modification of the internal architecture of a transformer layer. The proposed model places the multi-head attention sublayer and the MLP sublayer parallel to each other. Our experiments show that this simple modification improves the performance of transformer networks in multiple tasks. Moreover, for the image classification task, we show that using neural ODE solvers with a sophisticated integration scheme further improves performance.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

27 of 31 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2018Neural Ordinary Differential Equationscited by 3,733
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
15 more
  1. 2016Deep Networks with Stochastic Depthcited by 973
  2. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  3. 2020A Survey on Visual Transformercited by 653
  4. 2017Stable Architectures for Deep Neural Networkscited by 340
  5. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  6. 2018Scaling Neural Machine Translationcited by 204
  7. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  8. 2020Dissecting Neural ODEscited by 150
  9. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  10. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  11. 2022ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformerscited by 70
  12. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  13. 2021ViDT: An Efficient and Effective Fully Transformer-based Object Detectorcited by 45
  14. 2020"Hey, that's not an ODE": Faster ODE Adjoints via Seminormscited by 35
  15. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 5; 2026: 2.

13 papers.

Sort:

  1. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 12
  2. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  3. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  4. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  5. 2024Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODEcited by 6
  6. 2024ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergencecited by 5
  7. 2025Optimal and Diffusion Transports in Machine Learningcited by 5
  8. 2024Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breakingcited by 4
  9. 2024ICODE: Modeling Dynamical Systems with Extrinsic Input Informationcited by 1
  10. 2023Interpretation of the Transformer and Improvement of the Extractorcited by 0
  11. 2025ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equationcited by 0
  12. 2026SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifoldscited by 0
1 more
  1. 2026ODE-Based Transformer Decoders for Iterative Sign Language Translationcited by 0