mlsanityData to 14 Oct 2026

LightSeq2: Accelerated Training for Transformer-based Models on GPUs

arXiv 2110.05722 ↗First posted 12 Oct 2021cs.CL cs.MScited by 19

Not on the map. Why?

Abstract (arXiv)

Transformer-based neural models are used in many AI applications. Training these models is expensive, as it takes huge GPU resources and long duration. It is challenging because typical data like sentences have variable lengths, and Transformer's computation patterns are more complex than convolutional neural networks. Existing systems either only focus on model inference or optimization for only BERT-like encoder models. In this paper, we present LightSeq2, a system to accelerate training for a general family of Transformer models on GPUs. We propose a series of GPU optimization techniques tailored to the specific computation flow and memory access patterns of Transformer models. LightSeq2 supports many model architectures, including BERT (encoder-only), GPT (decoder-only), Transformer (encoder-decoder), and vision Transformer. Our experiments for a variety of models and benchmarks show that LightSeq2 is consistently faster (1.4-3.5x) than previous systems on different GPUs. In particular, it gains 308% training speedup compared with existing systems on a large public machine translation benchmark (WMT14 English-German).

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

27 of 43 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
15 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2021Scaling Vision Transformerscited by 999
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  5. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  6. 2021Random Feature Attentioncited by 354
  7. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  8. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  9. 2020Fast Transformers with Clustered Attentioncited by 123
  10. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  11. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  12. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  13. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  14. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  15. 2021LazyFormer: Self Attention with Lazy Updatecited by 16

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 6; 2024: 4; 2025: 2.

19 papers.

Sort:

By area:

  1. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  2. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  3. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  4. 2024A Survey on Hardware Accelerators for Large Language Modelscited by 16
  5. 2022FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hourscited by 14
  6. 2022HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddlecited by 12
  7. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8
  8. 2023NNQS-Transformer: an Efficient and Scalable Neural Network Quantum States Approach for Ab initio Quantum Chemistrycited by 8
  9. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  10. 2024Hardware Acceleration of LLMs: A comprehensive survey and comparisoncited by 3
  11. 2022TgDLF2.0: Theory-guided deep-learning for electrical load forecasting via Transformer and transfer learningcited by 2
  12. 2022PARAGEN : A Parallel Generation Toolkitcited by 2
7 more
  1. 2023Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processorscited by 2
  2. 2025Combining the Best of Both Worlds: A Method for Hybrid NMT and LLM Translationcited by 2
  3. 2025Two Heads Are Better than One: Simulating Large Transformers with Small Onescited by 2
  4. 2022Benchmark Assessment for DeepSpeed Optimization Librarycited by 1
  5. 2023Life Regression based Patch Slimming for Vision Transformerscited by 1
  6. 2022The VolcTrans System for WMT22 Multilingual Machine Translation Taskcited by 0
  7. 2024Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Trainingcited by 0