mlsanityData to 7 Oct 2026

ELLE: Efficient Lifelong Pre-training for Emerging Data

arXiv 2203.06311 ↗First posted 12 Mar 2022cs.CL cs.AIcited by 70

Not on the map. Why?

Abstract (arXiv)

Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. This requires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre-training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specifically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM's width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from 5 domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream performances. The codes are publicly available at https://github.com/thunlp/ELLE.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 12 Mar 2022.

Builds on

38 of 59 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2016Progressive Neural Networkscited by 1,332
26 more
  1. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2015Image-based Recommendations on Styles and Substitutescited by 702
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2019Green AIcited by 620
  9. 2019Defending Against Neural Fake Newscited by 553
  10. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  13. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  14. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  15. 2019Episodic Memory in Lifelong Language Learningcited by 188
  16. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  17. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  18. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  19. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  20. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  21. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  22. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  23. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  24. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  25. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  26. 2019Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brainscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 23; 2024: 20; 2025: 13; 2026: 4.

70 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023Continual Pre-training of Language Modelscited by 194
  4. 2023Mitigating the Alignment Tax of RLHFcited by 181
  5. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  6. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 174
  7. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  8. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  9. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  10. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  11. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  12. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
38 more
  1. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  2. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  3. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  4. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  5. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  6. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  7. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  8. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  9. 2022Exploring Mode Connectivity for Pre-trained Language Modelscited by 26
  10. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  11. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  12. 2023LEMON: Lossless model expansioncited by 23
  13. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  14. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  15. 2023Recyclable Tuning for Continual Pre-trainingcited by 17
  16. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  17. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  18. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Modelscited by 16
  19. 2024MUSCLE: A Model Update Strategy for Compatible LLM Evolutioncited by 15
  20. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 13
  21. 2025How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularizationcited by 13
  22. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
  23. 2023Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memorycited by 12
  24. 2025Memorization vs. Reasoning: Updating LLMs with New Knowledgecited by 12
  25. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 11
  26. 2023Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Studycited by 11
  27. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  28. 2023Continual Learning Under Language Shiftcited by 9
  29. 2024Exploring Forgetting in Large Language Model Pre-Trainingcited by 9
  30. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  31. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  32. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  33. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  34. 2025UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Modelscited by 7
  35. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  36. 2023AcademicGPT: Empowering Academic Researchcited by 6
  37. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  38. 2022PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in Englishcited by 5

50 most cited of 70 All 70 (JSON)