mlsanityData to 7 Oct 2026

Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning

arXiv 2604.01152 ↗First posted 1 Apr 2026cs.CL cs.AIcited by 1

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 22 of its 32 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

22 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  2. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin15 Dec
  3. 2023SiRA: Sparse Mixture of Low Rank Adaptation15 Nov

Other references · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2022Toy Models of Superpositioncited by 928
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  11. 2021Gradient Projection Memory for Continual Learningcited by 338
  12. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
7 more
  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  2. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  3. 2025Overcoming catastrophic forgetting in neural networkscited by 52
  4. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  5. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  6. 2025C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Modelscited by 2
  7. 2025Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Simulator-Grounded Large Language Models for Industrial Causal Reasoning: Tool-Use, Structured Injection, and Plant-Portable Retrieval for Wastewater Treatment Decision Supportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Routing-Based Continual Learning for Multimodal Large Language Models
  2. 2024Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment
  3. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  4. 2025StatLLaMA: Multi-Stage training for domain-optimized statistical large language models
  5. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
  6. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
  7. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
  8. 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios