mlsanityData to 7 Oct 2026

PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning

arXiv 2407.21571 ↗First posted 31 Jul 2024cs.CL cs.AIcited by 8

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 37 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

29 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2016Learning without Forgettingcited by 2,465
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
15 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  7. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  9. 2019Episodic Memory in Lifelong Language Learningcited by 188
  10. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  11. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  12. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  13. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 73
  14. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  15. 2023Investigating Forgetting in Pre-Trained Representations Through Continual Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  3. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  4. 2025Mixture-of-Experts for Personalized and Semantic-Aware Next Location Predictioncited by 2
  5. 2026Self-evolving Embodied AIcited by 2
  6. 2025Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learningcited by 1
  7. 2025EWC-Guided Diffusion Replay for Exemplar-Free Continual Learning in Medical Imagingcited by 0
  8. 2026RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
  2. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  3. 2025MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
  4. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
  5. 2026DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
  6. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
  7. 2026Mixture of Experts for Low-Resource LLMs
  8. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference