mlsanityData to 14 Oct 2026

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

arXiv 2610.13215 ↗First posted 9 Oct 2026cs.LGcited by 0

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 13 of its 13 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

13 of 13 references.

References · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  6. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  7. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  8. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 61
  9. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answeringcited by 37
  10. 2025HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Modelcited by 31
  11. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 13
  12. 2025Federated Continual Instruction Tuningcited by 12
1 more
  1. 2024MoRe Fine-Tuning with 10x Fewer Parameterscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
  2. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
  3. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
  4. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
  5. 2026PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
  6. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
  7. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  8. 2024SwitchCIT: Switching for Continual Instruction Tuning