Layer-Parallel Training for Transformers
Not on the map. Why?
Abstract (arXiv)
We present a new training methodology for transformers using a multilevel, layer-parallel approach. Through a neural ODE formulation of transformers, our application of a multilevel parallel-in-time algorithm for the forward and backpropagation phases of training achieves parallel acceleration over the layer dimension. This dramatically enhances parallel scalability as the network depth increases, which is particularly useful for increasingly large foundational models. However, achieving this introduces errors that cause systematic bias in the gradients, which in turn reduces convergence when closer to the minima. We develop an algorithm to detect this critical transition and either switch to serial training or systematically increase the accuracy of layer-parallel training. Results, including BERT, GPT2, ViT, and machine translation architectures, demonstrate parallel-acceleration as well as accuracy commensurate with serial pre-training while fine-tuning is unaffected.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 13 Jan 2026.
Builds on
30 of 57 references.
References · 30
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
18 more
- 2017Stable Architectures for Deep Neural Networkscited by 341
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
- 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
- 2018Marian: Fast Neural Machine Translation in C++cited by 203
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
- 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
- 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
- 2023A Guide Through the Zoo of Biased SGDcited by 40
- 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
- 2022On Optimizing the Communication of Model Parallelismcited by 19
- 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
- 2021ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translationcited by 12
- 2023DeepPCR: Parallelizing Sequential Operations in Neural Networkscited by 9
- 2023Breaking through the learning plateaus of in-context learning in Transformercited by 8
- 2019Multilevel Initialization for Layer-Parallel Deep Neural Network Trainingcited by 7
- 2022Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequencescited by 5
- 2021The Impact of Noise on Evaluation Complexity: The Deterministic Trust-Region Casecited by 4
- 2024Design Guidelines for Noise-Tolerant Optimization with Applications in Robust Designcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.