mlsanityData to 14 Oct 2026

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

arXiv 1908.08962 ↗First posted 23 Aug 2019cs.CLcited by 120

Not on the map. Why?

Abstract (arXiv)

Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 27; 2023: 28; 2024: 21; 2025: 12; 2026: 12.

120 papers.

Sort:

By area:

  1. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  2. 2021Block Pruning For Faster Transformerscited by 177
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  5. 2021Learning Compact Metrics for MTcited by 93
  6. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  7. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 59
  8. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  9. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  10. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 49
  11. 2024A Survey on Transformer Compressioncited by 46
  12. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
38 more
  1. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  2. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  3. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  4. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  5. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  6. 2022Composable Text Controls in Latent Space with ODEscited by 28
  7. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  8. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  9. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  10. 2023Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicinecited by 24
  11. 2022Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Settingcited by 23
  12. 2023A Survey on Contextualised Semantic Shift Detectioncited by 22
  13. 2023LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformerscited by 22
  14. 2022Weakly-supervised segmentation of referring expressionscited by 21
  15. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  16. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 17
  17. 2022Debiasing Masks: A New Framework for Shortcut Mitigation in NLUcited by 16
  18. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  19. 2021ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquescited by 14
  20. 2022Deep Learning for Hate Speech Detection: A Comparative Studycited by 13
  21. 2022Differentially Private Model Compressioncited by 13
  22. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  23. 2025Structured Agent Distillation for Large Language Modelcited by 13
  24. 2022FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?cited by 12
  25. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  26. 2025Learning from Diverse Reasoning Paths with Routing and Collaborationcited by 12
  27. 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11
  28. 2023Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Studycited by 11
  29. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  30. 2024ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generationcited by 11
  31. 2021Quantization-Guided Training for Compact TinyML Modelscited by 10
  32. 2022Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Trainingcited by 10
  33. 2022Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithmcited by 9
  34. 2021DoT: An efficient Double Transformer for NLP tasks with tablescited by 8
  35. 2021Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performancecited by 8
  36. 2021Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Modelscited by 7
  37. 2023Web Content Filtering through knowledge distillation of Large Language Modelscited by 7
  38. 2021SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensionscited by 6

50 most cited of 120 All 120 (JSON)