mlsanityData to 14 Oct 2026

DynaBERT: Dynamic BERT with Adaptive Width and Depth

arXiv 2004.04037 ↗First posted 8 Apr 2020cs.CL cs.LGcited by 251

Not on the map. Why?

Cited by 32 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The pre-trained language models like BERT, though powerful in many natural language processing tasks, are both computation and memory expensive. To alleviate this problem, one approach is to compress them for specific tasks before deployment. However, recent works on BERT compression usually compress the large BERT model to a fixed smaller size. They can not fully satisfy the requirements of different edge devices with various hardware performances. In this paper, we propose a novel dynamic BERT model (abbreviated as DynaBERT), which can flexibly adjust the size and latency by selecting adaptive width and depth. The training process of DynaBERT includes first training a width-adaptive BERT and then allowing both adaptive width and depth, by distilling knowledge from the full-sized model to small sub-networks. Network rewiring is also used to keep the more important attention heads and neurons shared by more sub-networks. Comprehensive experiments under various efficiency constraints demonstrate that our proposed dynamic BERT (or RoBERTa) at its largest size has comparable performance as BERT-base (or RoBERTa-base), while at smaller widths and depths consistently outperforms existing BERT compression methods. Code is available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT.

Key facts. Cited by 251 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 37; 2022: 56; 2023: 52; 2024: 53; 2025: 25; 2026: 28.

251 papers.

Sort:

By area:

  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  2. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  4. 2022Confident Adaptive Language Modelingcited by 310
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  9. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  10. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  11. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
38 more
  1. 2021Learned Token Pruning for Transformerscited by 139
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  3. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  4. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  5. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 115
  6. 2021SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Predictioncited by 97
  7. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 93
  8. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  9. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  10. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 85
  11. 2021Dynamic Slimmable Networkcited by 83
  12. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  13. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  14. 2022Unified Visual Transformer Compressioncited by 79
  15. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  16. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  17. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  18. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  19. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  20. 2024Dynamic Diffusion Transformercited by 62
  21. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  22. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  23. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  24. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
  25. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  26. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 49
  27. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
  28. 2024A Survey on Transformer Compressioncited by 46
  29. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 46
  30. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 46
  31. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  32. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  33. 2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimizationcited by 43
  34. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  35. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39
  36. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 37
  37. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 37
  38. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37

50 most cited of 251 All 251 (JSON)