mlsanityData to 14 Oct 2026

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers

arXiv 2602.12587 ↗First posted 13 Feb 2026cs.LGcited by 1

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number Neff and find that higher Neff is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 24 of its 29 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

24 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Self-Evolving LLMs via Continual Instruction Tuning14 Sep
  2. 2024Theory on Mixture-of-Experts in Continual Learning24 Jun

Other references · 22

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  10. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
10 more
  1. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  2. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  3. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  4. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  5. 2023Continual Pre-training of Language Modelscited by 198
  6. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  7. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 76
  8. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 66
  9. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  10. 2024MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgettingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learningcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2026Attention-Aware Routing: Coupling Routing and Attention in MoEs
  2. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token
  3. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  4. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
  5. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
  6. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
  7. 2026PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
  8. 2026Mixture of Layers with Hybrid Attention