mlsanityData to 7 Oct 2026

μ-Parametrization for Mixture of Experts

arXiv 2508.09752 ↗First posted 13 Aug 2025cs.LGcited by 3

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Recent years have seen a growing interest and adoption of LLMs, with Mixture-of-Experts (MoE) emerging as a leading architecture in extremely large models. Currently, the largest open-source models reach over 1T parameters. At such scales, hyperparameter tuning becomes prohibitively expensive. Precisely for this reason, the μTransfer is becoming a key technique. It allows for seamless transfer of optimal hyperparameters across model scales, resulting in a huge reduction in tuning costs. However, existing work has primarily focused on dense LLMs, leaving MoE architectures unexplored. In this work, we derive a μ-Parameterization for MoE, providing theoretical guarantees for feature learning across model widths. Our experiments demonstrate that the optimal learning rate reliably transfers across model sizes, establishing a foundation for efficient hyperparameter tuning in large-scale MoE models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 14 of its 18 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

14 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient7 Feb

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  9. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  10. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  11. 2023A Spectral Condition for Feature Learningcited by 101
  12. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
1 more
  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Modelscited by 1
  2. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterizationcited by 0
  3. 2026Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hyperparameter Transfer with Mixture-of-Expert Layers
  2. 2026Hyperparameter Scaling Laws Across MoE Sparsity
  3. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  4. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  6. 2026ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
  7. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  8. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models