mlsanityData to 7 Oct 2026

Reducing Transformer Depth on Demand with Structured Dropout

arXiv 1909.11556 ↗First posted 25 Sep 2019cs.LG cs.CL stat.MLcited by 434

Not on the map. Why?

Cited by 46 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality compared to training from scratch or using distillation.

Key facts. Cited by 434 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 78; 2022: 72; 2023: 82; 2024: 74; 2025: 56; 2026: 72.

434 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  4. 2021Going deeper with Image Transformerscited by 759
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  9. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
  10. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
38 more
  1. 2022Confident Adaptive Language Modelingcited by 297
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  3. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  4. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  5. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  6. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  8. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 194
  9. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
  10. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  11. 2021Block Pruning For Faster Transformerscited by 176
  12. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  13. 2023Speculative Decoding with Big Little Decodercited by 163
  14. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  15. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  16. 2022FlexiViT: One Model for All Patch Sizescited by 151
  17. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  18. 2021Learned Token Pruning for Transformerscited by 138
  19. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138
  20. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  21. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  22. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  23. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  24. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  25. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  26. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  27. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  28. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  29. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  30. 2021Finetuning Pretrained Transformers into RNNscited by 94
  31. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  32. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  33. 2022Unified Visual Transformer Compressioncited by 79
  34. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  35. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  36. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  37. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  38. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72

50 most cited of 434 All 434 (JSON)