mlsanityData to 7 Oct 2026

A Survey on Efficient Training of Transformers

arXiv 2302.01107 ↗First posted 2 Feb 2023cs.LG cs.AI cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 81 of its 102 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

81 of 102 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  13. 2017Mixed Precision Trainingcited by 1,135
  14. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  15. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  16. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  17. 2021Going deeper with Image Transformerscited by 759
  18. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  19. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  20. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  21. 2019Green AIcited by 620
  22. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  23. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  24. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  25. 2017Large Batch Training of Convolutional Networkscited by 529
  26. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  27. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  28. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  29. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  30. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  31. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  32. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  33. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  34. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  35. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  36. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  37. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  38. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 10; 2025: 10; 2026: 5.

37 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  5. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  6. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  7. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  8. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  9. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  10. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  11. 2023Towards Diverse Binary Segmentation via A Simple yet General Gated Networkcited by 17
  12. 2024ToDo: Token Downsampling for Efficient Generation of High-Resolution Imagescited by 14
25 more
  1. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  2. 2023Fauno: The Italian Large Language Model that will leave you senza parole!cited by 13
  3. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  4. 2024A Survey on Structure-Preserving Graph Transformerscited by 7
  5. 2025Accelerating Training Speed of Tiny Recursive Models with Curriculum Guided Adaptive Recursioncited by 5
  6. 2024Optimizing RAG Techniques for Automotive Industry PDF Chatbots: A Case Study with Locally Deployed Ollama Modelscited by 4
  7. 2024Does Self-Attention Need Separate Weights in Transformers?cited by 4
  8. 2024Continuous Speech Tokenizer in Text To Speechcited by 3
  9. 2024Signformer is all you need: Towards Edge AI for Sign Languagecited by 3
  10. 2025AI for Regulatory Affairs: Balancing Accuracy, Interpretability, and Computational Cost in Medical Device Classificationcited by 2
  11. 2024KEDformer:Knowledge Extraction Seasonal Trend Decomposition for Long-term Sequence Predictioncited by 1
  12. 2025Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justicecited by 1
  13. 2025Agentic Physical AI toward a Domain-Specific Foundation Model for Energy Systems: A Case Study on Nuclear Reactor Controlcited by 1
  14. 2024Federating to Grow Transformers with Constrained Resources without Model Sharingcited by 0
  15. 2025Deeply Dual Supervised learning for melanoma recognitioncited by 0
  16. 2025Edge Detection for Organ Boundaries via Top Down Refinement and SubPixel Upsamplingcited by 0
  17. 2025DualResolution Residual Architecture with Artifact Suppression for Melanocytic Lesion Segmentationcited by 0
  18. 2025VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagationcited by 0
  19. 2025Deceptive Risk Minimization: Out-of-Distribution Generalization by Deceiving Distribution Shift Detectorscited by 0
  20. 2025Enhancing Time-Series Anomaly Detection by Integrating Spectral-Residual Bottom-Up Attention with Reservoir Computingcited by 0
  21. 2026Rethinking Recurrent Neural Networks for Time Series Forecasting: A Reinforced Recurrent Encoder with Prediction-Oriented Proximal Policy Optimizationcited by 0
  22. 2026Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scalingcited by 0
  23. 2026Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentationcited by 0
  24. 2026Imputation Meets Clustering: Exploiting Latent Subgroup Structure for Missing Data Recoverycited by 0
  25. 2026MixFormer: Linear Transformer with Mixture of Memory Expertscited by 0