mlsanityData to 7 Oct 2026

Lifelong Language Pretraining with Distribution-Specialized Experts

arXiv 2305.12281 ↗First posted 20 May 2023cs.CL cs.LGcited by 64

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Pretraining on a large-scale corpus has become a standard method to build general language models (LMs). Adapting a model to new data distributions targeting different downstream tasks poses significant challenges. Naive fine-tuning may incur catastrophic forgetting when the over-parameterized LMs overfit the new data but fail to preserve the pretrained features. Lifelong learning (LLL) aims to enable information systems to learn from a continuous data stream across time. However, most prior work modifies the training recipe assuming a static fixed network architecture. We find that additional model capacity and proper regularization are key elements to achieving strong LLL performance. Thus, we propose Lifelong-MoE, an extensible MoE (Mixture-of-Experts) architecture that dynamically adds model capacity via adding experts with regularized pretraining. Our results show that by only introducing a limited number of extra experts while keeping the computation cost constant, our model can steadily adapt to data distribution shifts while preserving the previous knowledge. Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 49 of its 63 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

49 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  2. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
33 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2016Learning without Forgettingcited by 2,465
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2017Continual Learning with Deep Generative Replaycited by 1,152
  8. 2018Experience Replay for Continual Learningcited by 1,074
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  12. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  13. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  14. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  15. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  16. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  17. 2015Skip-Thought Vectorscited by 321
  18. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  19. 2015Semi-supervised Sequence Learningcited by 198
  20. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  21. 2019Episodic Memory in Lifelong Language Learningcited by 188
  22. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  23. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  24. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  25. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  26. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  27. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  28. 2019Sentence Embedding Alignment for Lifelong Relation Extractioncited by 70
  29. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
  30. 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
  31. 2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
  32. 2020Meta-Learning with Sparse Experience Replay for Lifelong Language Learningcited by 12
  33. 2021DRILL: Dynamic Representations for Imbalanced Lifelong Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 27; 2025: 20; 2026: 12.

64 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2023Mitigating the Alignment Tax of RLHFcited by 181
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  7. 2024Mixture of A Million Expertscited by 70
  8. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  9. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  10. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  11. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  12. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
38 more
  1. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  2. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  3. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  4. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  5. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  6. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  7. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  8. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  9. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  10. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  11. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 13
  12. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  13. 2024Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligencecited by 11
  14. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  15. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  16. 2025Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detectioncited by 8
  17. 2023Continual Learning with Dirichlet Generative-based Rehearsalcited by 7
  18. 2024MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgettingcited by 6
  19. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  20. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  21. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  22. 2024VA-MoE: Variables-Adaptive Mixture of Experts for Incremental Weather Forecastingcited by 5
  23. 2024Continual learning with the neural tangent ensemblecited by 4
  24. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
  25. 2025Learning Dynamics in Continual Pre-Training for Large Language Modelscited by 4
  26. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  27. 2025Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Modelscited by 4
  28. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  29. 2024Auto-selected Knowledge Adapters for Lifelong Person Re-identificationcited by 3
  30. 2024Scalable Multi-Domain Adaptation of Language Models using Modular Expertscited by 3
  31. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
  32. 2025Lifelong Learner: Discovering Versatile Neural Solvers for Vehicle Routing Problemscited by 2
  33. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  34. 2024Drift to Remembercited by 1
  35. 2025R^2MoE: Redundancy-Removal Mixture of Experts for Lifelong Concept Learningcited by 1
  36. 2025Analyzing and Internalizing Complex Policy Documents for LLM Agentscited by 1
  37. 2026Continual Policy Consolidation for Lifelong Robot Learningcited by 1
  38. 2026Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformerscited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmap
  3. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
  4. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  5. 2026Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
  6. 2025From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs
  7. 2024Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning
  8. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts