Data to 7 Oct 2026

Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping

arXiv 2010.13369 ↗First posted 26 Oct 2020cs.LGcited by 93

Not on the map. Why?

Abstract (arXiv)

Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains. However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses. Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling. In this work, we propose a method based on progressive layer dropping that speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency. Extensive experiments on BERT show that the proposed method achieves a 24% time reduction on average per sample and allows the pre-training to be 2.5 times faster than the baseline to get a similar accuracy on downstream tasks. While being faster, our pre-trained models are equipped with strong knowledge transferability, achieving comparable and sometimes higher GLUE score than the baseline when pre-trained with the same number of samples.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 16; 2023: 19; 2024: 33; 2025: 9; 2026: 5.

93 papers.

Sort:

By area:

2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
2021Pre-Trained Models: Past, Present and Futurecited by 319
2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
2021How to Train BERT with an Academic Budgetcited by 80
2022Unified Visual Transformer Compressioncited by 79

38 more

2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimizationcited by 43
2022Token Dropping for Efficient BERT Pretrainingcited by 42
2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 42
2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
2023A Survey on Efficient Training of Transformerscited by 37
2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 35
2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
2024Data curation via joint example selection further accelerates multimodal learningcited by 32
2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
2023Federated Learning-Empowered AI-Generated Content in Wireless Networkscited by 26
20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
2021OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Servicescited by 20
2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 19
2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
2024SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezingcited by 12
2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
2023Revisiting Token Dropping Strategy in Efficient BERT Pretrainingcited by 11
2024Persistent Topological Features in Large Language Modelscited by 11
2022Progressive Transformation Learning for Leveraging Virtual Images in Trainingcited by 10
2024Preparing Lessons for Progressive Training on Language Modelscited by 10
2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
2022Graph Reasoning Transformer for Image Parsingcited by 9

50 most cited of 93 All 93 (JSON)