Data to 7 Oct 2026

On the Effect of Dropping Layers of Pre-trained Transformer Models

arXiv 2004.03844 ↗First posted 8 Apr 2020cs.CL cs.LGcited by 118

Not on the map. Why?

Abstract (arXiv)

Transformer-based NLP models are trained using hundreds of millions or even billions of parameters, limiting their applicability in computationally constrained environments. While the number of parameters generally correlates with performance, it is not clear whether the entire network is required for a downstream task. Motivated by the recent work on pruning and distilling pre-trained models, we explore strategies to drop layers in pre-trained models, and observe the effect of pruning on downstream GLUE tasks. We were able to prune BERT, RoBERTa and XLNet models up to 40%, while maintaining up to 98% of their original performance. Additionally we show that our pruned models are on par with those built using knowledge distillation, both in terms of size and performance. Our experiments yield interesting observations such as, (i) the lower layers are most critical to maintain downstream task performance, (ii) some tasks such as paraphrase detection and sentence similarity are more robust to the dropping of layers, and (iii) models trained using a different objective function exhibit different learning patterns and w.r.t the layer dropping.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 24; 2024: 38; 2025: 25; 2026: 16.

118 papers.

Sort:

By area:

2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2021Learned Token Pruning for Transformerscited by 138
2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
2021A Practical Survey on Faster and Lighter Transformerscited by 73
2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
2021Differentiable Subset Pruning of Transformer Headscited by 46

38 more

2023Discrete Prompt Compression with Reinforcement Learningcited by 42
2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
2024A Survey of Small Language Modelscited by 31
2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
2023Adapting a Language Model While Preserving its General Knowledgecited by 23
2022Discovering Salient Neurons in Deep NLP Modelscited by 21
2023Fairness-Aware Structured Pruning in Transformerscited by 21
2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
2024OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimizationcited by 17
2024FedPFT: Federated Proxy Fine-Tuning of Foundation Modelscited by 16
2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformerscited by 16
2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
2024Probing Multimodal Large Language Models for Global and Local Semantic Representationscited by 12
2023Accurate Retraining-free Pruning for Pretrained Encoder-based Language Modelscited by 11
2024Persistent Topological Features in Large Language Modelscited by 11
2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMscited by 11
2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
2025Dr.LLM: Dynamic Layer Routing in LLMscited by 10
2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
2026ERNIE 5.0 Technical Reportcited by 9
2024Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similaritycited by 8
2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
2024Hierarchical Skip Decoding for Efficient Autoregressive Text Generationcited by 7
2024Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Modelscited by 7
2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusioncited by 6
2024Large Language Models Are Overparameterized Text Encoderscited by 6

50 most cited of 118 All 118 (JSON)