mlsanityData to 7 Oct 2026

TinyBERT: Distilling BERT for Natural Language Understanding

arXiv 1909.10351 ↗First posted 23 Sep 2019cs.CL cs.AI cs.LGcited by 1,153

Not on the map. Why?

Cited by 41 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be effectively transferred to a small student Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture he general-domain as well as the task-specific knowledge in BERT. TinyBERT with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT with 4 layers is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them. Moreover, TinyBERT with 6 layers performs on-par with its teacher BERTBASE.

Key facts. Cited by 1,153 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 204; 2022: 217; 2023: 249; 2024: 161; 2025: 166; 2026: 156.

1,153 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  10. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  11. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  12. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
38 more
  1. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2022Confident Adaptive Language Modelingcited by 297
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  5. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  6. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  7. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  8. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  9. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  10. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  11. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  12. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  13. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  14. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  15. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  16. 2022Language model compression with weighted low-rank factorizationcited by 192
  17. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  18. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  19. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  20. 2021Block Pruning For Faster Transformerscited by 176
  21. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  22. 2023Speculative Decoding with Big Little Decodercited by 163
  23. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  24. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  25. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  26. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  27. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  28. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  29. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  30. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  31. 2021Learned Token Pruning for Transformerscited by 138
  32. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  33. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  34. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  35. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  36. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  37. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  38. 2022Pretrained Language Models for Text Generation: A Surveycited by 121

50 most cited of 1,153 All 1,153 (JSON)