mlsanityData to 7 Oct 2026

Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models

arXiv 2201.00558 ↗First posted 3 Jan 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in two tasks: text classification and sequence labeling in the Indonesian language. We also compare various aspects of distillations including the usage of word embeddings and unlabeled data augmentation. Our experiments show that, despite the rising popularity of Transformer-based models, using BiLSTM and CNN student models provide the best trade-off between performance and computational resource (CPU, RAM, and storage) compared to pruned BERT models. We further propose some quick wins on performing KD to produce small NLP models via efficient KD training mechanisms involving simple choices of loss functions, word embeddings, and unlabeled data preparation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

42 of 59 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
30 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2013Network In Networkcited by 600
  8. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  11. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  12. 2015Semi-supervised Sequence Learningcited by 198
  13. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  14. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  15. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  16. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  17. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  18. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  19. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  20. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
  21. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  22. 2019Unsupervised Model Selection for Variational Disentangled Representation Learningcited by 46
  23. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  24. 2019Unifying Heterogeneous Classifiers with Distillationcited by 24
  25. 2020Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Functioncited by 23
  26. 2020Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vectorcited by 18
  27. 2021Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processingcited by 7
  28. 2020Investigating Bi-LSTM and CRF with POS Tag Embedding for Indonesian Named Entity Taggercited by 5
  29. 2018Toward a Standardized and More Accurate Indonesian Part-of-Speech Taggingcited by 3
  30. 2019Aspect and Opinion Terms Extraction Using Double Embeddings and Attention Mechanism for Indonesian Hotel Reviewscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2024The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillationcited by 4
  2. 2025Small Language Models in the Real World: Insights from Industrial Text Classificationcited by 2
  3. 2023Improving Neural Topic Models with Wasserstein Knowledge Distillationcited by 1
  4. 2024Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Servicecited by 0
  5. 2025On Multilingual Encoder Language Model Compression for Low-Resource Languagescited by 0