mlsanityData to 7 Oct 2026

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

arXiv 1910.01108 ↗First posted 2 Oct 2019cs.CLcited by 4,243

Not on the map. Why?

Cited by 133 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

Key facts. Cited by 4,243 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 531; 2022: 654; 2023: 853; 2024: 852; 2025: 776; 2026: 575.

4,243 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2022Efficiently Scaling Transformer Inferencecited by 529
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  13. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  14. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  16. 2022Memory-Based Model Editing at Scalecited by 443
  17. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  18. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  19. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  20. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  21. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  22. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  23. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  24. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  25. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  26. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  27. 2021Random Feature Attentioncited by 350
  28. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  29. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  30. 2021True Few-Shot Learning with Language Modelscited by 343
  31. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  32. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  33. 2022Large Language Models Are Reasoning Teacherscited by 318
  34. 2022Confident Adaptive Language Modelingcited by 297
  35. 2022UL2: Unifying Language Learning Paradigmscited by 292
  36. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  37. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  38. 2026Reinforcement Learning via Self-Distillationcited by 285

50 most cited of 4,243 All 4,243 (JSON)