mlsanityData to 14 Oct 2026

A multilevel approach to accelerate the training of Transformers

arXiv 2504.18590 ↗First posted 24 Apr 2025cs.LG cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In this article, we investigate the potential of multilevel approaches to accelerate the training of transformer architectures. Using an ordinary differential equation (ODE) interpretation of these architectures, we propose an appropriate way of varying the discretization of these ODE Transformers in order to accelerate the training. We validate our approach experimentally by a comparison with the standard training procedure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

22 of 29 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  8. 2020Understanding the Difficulty of Training Transformerscited by 210
  9. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  10. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  11. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  12. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
10 more
  1. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  2. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 27
  3. 2020Exploring Transformers for Large-Scale Speech Recognitioncited by 24
  4. 2020Multilevel Minimization for Deep Residual Networkscited by 11
  5. 2021Globally Convergent Multilevel Training of Deep Residual Networkscited by 11
  6. 2019Neural ODEs as the Deep Limit of ResNets with constant weightscited by 10
  7. 2020N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equationscited by 9
  8. 2021Speeding up Deep Model Training by Sharing Weights and Then Unsharingcited by 7
  9. 2023A Block-Coordinate Approach of Multi-level Optimization with an Application to Physics-Informed Neural Networkscited by 7
  10. 2023IML FISTA: A Multilevel Framework for Inexact and Inertial Forward-Backward. Application to Image Restorationcited by 2

Built on by

Not cited yet.