mlsanityData to 7 Oct 2026

Distilling Task-Specific Knowledge from BERT into Simple Neural Networks

arXiv 1903.12136 ↗First posted 28 Mar 2019cs.CL cs.LGcited by 152

Not on the map. Why?

Abstract (arXiv)

In the natural language processing literature, neural networks are becoming increasingly deeper and complex. The recent poster child of this trend is the deep language representation model, which includes BERT, ELMo, and GPT. These developments have led to the conviction that previous-generation, shallower neural networks for language understanding are obsolete. In this paper, however, we demonstrate that rudimentary, lightweight neural networks can still be made competitive without architecture changes, external training data, or additional input features. We propose to distill knowledge from BERT, a state-of-the-art language representation model, into a single-layer BiLSTM, as well as its siamese counterpart for sentence-pair tasks. Across multiple datasets in paraphrasing, natural language inference, and sentiment classification, we achieve comparable results with ELMo, while using roughly 100 times fewer parameters and 15 times less inference time.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 40; 2022: 24; 2023: 35; 2024: 30; 2025: 15; 2026: 8.

152 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  3. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2023Pretraining Language Models with Human Preferencescited by 245
  6. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  7. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  8. 2021Block Pruning For Faster Transformerscited by 176
  9. 2023Speculative Decoding with Big Little Decodercited by 163
  10. 2021Learned Token Pruning for Transformerscited by 138
  11. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  12. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
38 more
  1. 2021Trustworthy AI: A Computational Perspectivecited by 93
  2. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  6. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  7. 2024A Survey on Transformer Compressioncited by 44
  8. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  9. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  10. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 39
  11. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  12. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  13. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  14. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  15. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  16. 2023Embedding Democratic Values into Social Media AIs via Societal Objective Functionscited by 25
  17. 2021Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapterscited by 23
  18. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  19. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
  20. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  21. 2023Mixed Distillation Helps Smaller Language Model Better Reasoningcited by 18
  22. 2021NewsBERT: Distilling Pre-trained Language Model for Intelligent News Applicationcited by 17
  23. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architecturescited by 16
  24. 2021Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIscited by 16
  25. 2023Cross-modality Data Augmentation for End-to-End Sign Language Translationcited by 16
  26. 2023Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognitioncited by 16
  27. 2023SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillationcited by 15
  28. 2021ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquescited by 14
  29. 2021Elbert: Fast Albert with Confidence-Window Based Early Exitcited by 14
  30. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
  31. 2022Adaptable Adapterscited by 13
  32. 2022Differentially Private Model Compressioncited by 13
  33. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  34. 2023LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillationcited by 11
  35. 2023Accurate Retraining-free Pruning for Pretrained Encoder-based Language Modelscited by 11
  36. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 11
  37. 2025TwT: Thinking without Tokens by Habitual Reasoning Distillation with Multi-Teachers' Guidancecited by 11
  38. 2021Distilling Large Language Models into Tiny and Effective Students using pQRNNcited by 10

50 most cited of 152 All 152 (JSON)