mlsanityData to 14 Oct 2026

UniPool: A Globally Shared Expert Pool for Mixture-of-Experts

arXiv 2605.06665 ↗First posted 7 May 2026cs.LG cs.AIcited by 1

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-to-layer allocation that is then executed privately. UniPool-Lock gives every layer its own router over one global pool, balances aggregate pool usage, and scores experts with a scale-stable NormRouter. After the short ownership-learning phase, it assigns each expert to one layer, locks the disjoint allocation, and continues training as a layer-private MoE. With the same expert-FFN budget and routed expert FLOPs, this recipe lowers held-out loss relative to vanilla MoE by 0.024-0.037 across five dense-equivalent scales from 182M to 1.5B, including -0.0247 at 1.5B after 60B tokens. The ownership-learning phase lasts 2K steps, about 3.3% of training, and post-lock step time is within -0.7% to +2.8% of vanilla MoE in our throughput measurements. Controls locate the gain in full-pool training and in the allocation it produces: a random disjoint allocation fixed at initialization, trained with the same router and losses, matches vanilla, whereas locking the allocation learned during the full-pool phase recovers nearly all of the persistent full-pool gain, and substituting a random allocation at the lock forfeits about half of it. Keeping full-pool access throughout training (UniPool-Full) also improves over vanilla MoE at four scales and outperforms it with only 66.7% (182M) to 50% (469M and 650M) of its expert parameters.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 46 of its 50 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

46 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation5 Mar
  2. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  3. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  4. 2024Mixture of A Million Experts4 Jul
  5. 2024MoEUT: Mixture-of-Experts Universal Transformers25 May
  6. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  7. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate29 Dec
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 38

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
26 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  4. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  5. 2018Universal Transformerscited by 647
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  10. 2021Hash Layers For Large Sparse Modelscited by 237
  11. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  12. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  13. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  14. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  15. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  16. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  17. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  18. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  19. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 42
  20. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 26
  21. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 24
  22. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  23. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Expertscited by 10
  24. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
  25. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Modelscited by 3
  26. 2026SD-MoE: Spectral Decomposition for Effective Expert Specializationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
  2. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  3. 2026MoRE: Mixture of Reused Experts
  4. 2026Towards a Statistical Understanding of Mixture-of-Experts
  5. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  6. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  7. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
  8. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling