mlsanityData to 14 Oct 2026

Knowledge Inheritance for Pre-trained Language Models

arXiv 2105.13880 ↗First posted 28 May 2021cs.CL cs.AI cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

46 of 70 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
34 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2017Mixed Precision Trainingcited by 1,156
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2019Green AIcited by 627
  13. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  14. 2018Born Again Neural Networkscited by 546
  15. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  16. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  17. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  18. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  19. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  20. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  21. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  22. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  23. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 139
  24. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  25. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  26. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  27. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  28. 2020ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learningcited by 72
  29. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  30. 2021Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Studycited by 51
  31. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  32. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  33. 2019Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraintscited by 37
  34. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 11; 2024: 7; 2025: 3; 2026: 5.

38 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  5. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  6. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  7. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  8. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  9. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  10. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  11. 2022Transformers in Time-series Analysis: A Tutorialcited by 46
  12. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 33
26 more
  1. 2023Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Modelscited by 31
  2. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  3. 2022Exploring Mode Connectivity for Pre-trained Language Modelscited by 26
  4. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  5. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  6. 2023LEMON: Lossless model expansioncited by 23
  7. 2023Recyclable Tuning for Continual Pre-trainingcited by 17
  8. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  9. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  10. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  11. 2024Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalizationcited by 10
  12. 2023A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Modelscited by 9
  13. 2025Efficient Construction of Model Family through Progressive Training Using Model Expansioncited by 8
  14. 2024mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysiscited by 5
  15. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  16. 2022Continual Knowledge Distillation for Neural Machine Translationcited by 4
  17. 2025Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Lawscited by 1
  18. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  19. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  20. 2026The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignmentcited by 1
  21. 2024Induced Model Matching: Restricted Models Help Train Full-Featured Modelscited by 0
  22. 2024A Multi-Level Framework for Accelerating Training Transformer Modelscited by 0
  23. 2024Federating to Grow Transformers with Constrained Resources without Model Sharingcited by 0
  24. 2026Ruyi2.5 Technical Reportcited by 0
  25. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  26. 2026Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration
  2. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  3. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  4. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Models
  5. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Models
  6. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation
  7. 2021Dynamic Knowledge Distillation for Pre-trained Language Models
  8. 2026Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails