mlsanityData to 7 Oct 2026

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation

arXiv 2603.04971 ↗First posted 5 Mar 2026cs.LG cs.AI cs.CLcited by 7

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 29 of its 42 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

29 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2024Mixtral of Expertscited by 1,785
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
15 more
  1. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  2. 2018Universal Transformerscited by 613
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2023From Sparse to Soft Mixtures of Expertscited by 245
  6. 2021Hash Layers For Large Sparse Modelscited by 236
  7. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  8. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  9. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  10. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  11. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  12. 2024Loop Neural Networks for Parameter Sharingcited by 10
  13. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 10
  14. 2025DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Expertscited by 7
  15. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  2. 2026LT2: Linear-Time Looped Transformerscited by 5
  3. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Modelscited by 2
  4. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Expertscited by 1
  5. 2026MoRE: Mixture of Reused Expertscited by 1
  6. 2026Mixture of Layers with Hybrid Attentioncited by 0
  7. 2026How to Loop MoE: Flatten the Experts, Untie the Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  2. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
  3. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
  4. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  5. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  6. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  7. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
  8. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework