# Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

arXiv: https://arxiv.org/abs/1908.08962 · HTML: https://mlsanity.com/paper/1908.08962/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2019.

## Built on by (most cited first, 30 of 120)

- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models](https://mlsanity.com/paper/2203.07259/) (2022)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [BERT Learns to Teach: Knowledge Distillation with Meta Learning](https://mlsanity.com/paper/2106.04570/) (2021)
- [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021)
- [Outliers Dimensions that Disrupt Transformers Are Driven by Frequency](https://mlsanity.com/paper/2205.11380/) (2022)
- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [ZipLM: Inference-Aware Structured Pruning of Language Models](https://mlsanity.com/paper/2302.04089/) (2023)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [Towards Efficient NLP: A Standard Evaluation and A Strong Baseline](https://mlsanity.com/paper/2110.07038/) (2021)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression](https://mlsanity.com/paper/2109.03228/) (2021)
- [Dynamic Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2109.11295/) (2021)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)
- [Composable Text Controls in Latent Space with ODEs](https://mlsanity.com/paper/2208.00638/) (2022)
- [Lifting the Curse of Capacity Gap in Distilling Language Models](https://mlsanity.com/paper/2305.12129/) (2023)
- [How Does Adaptive Optimization Impact Local Neural Network Geometry?](https://mlsanity.com/paper/2211.02254/) (2022)
- [AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers](https://mlsanity.com/paper/2302.14705/) (2023)
- [Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine](https://mlsanity.com/paper/2311.04937/) (2023)
- [Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting](https://mlsanity.com/paper/2203.07856/) (2022)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers](https://mlsanity.com/paper/2305.18396/) (2023)
- [Weakly-supervised segmentation of referring expressions](https://mlsanity.com/paper/2205.04725/) (2022)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Debiasing Masks: A New Framework for Shortcut Mitigation in NLU](https://mlsanity.com/paper/2210.16079/) (2022)
- [GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model](https://mlsanity.com/paper/2306.06629/) (2023)

## Abstract (arXiv)

Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.