mlsanityData to 14 Oct 2026

Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation

arXiv 2610.14644 ↗First posted 10 Oct 2026cs.AI cs.LGcited by 0

On the map: Mixture-of-experts → Expert routing strategies also fits: MoE theory and gating functions

Abstract (arXiv)

Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity. LatentMoE moves expert computation into a lower-dimensional space, exposing the input dimension and intermediate width of each expert as explicit design choices. We show that an expert's local Jacobian rank is jointly bounded by these two dimensions, and that balancing them maximizes the rank ceiling at fixed per-expert parameters. Controlled shape sweeps across model scales and routing granularities consistently favor near-square experts. We further use layerwise intrinsic dimension (ID) to guide capacity allocation across depth, co-varying the number and shape of experts while keeping per-layer parameter count, activated FLOPs, and communication volume constant. Compared with layer-homogeneous baselines, our ID-guided allocation achieves lower validation loss and stronger downstream performance. These results establish expert shape and layerwise allocation as practical design axes for MoEs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 27 of its 30 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

27 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation15 Aug
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2024Mixtral of Expertscited by 1,823
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  9. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  10. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  11. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  12. 2026Kimi K3: Open Frontier Intelligencecited by 138
13 more
  1. 2023The geometry of hidden representations of large transformer modelscited by 112
  2. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  3. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 29
  4. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 14
  5. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translationcited by 10
  6. 2025MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Modelscited by 10
  7. 2025Geometry of Decision Making in Language Modelscited by 9
  8. 2024Mixture of Diverse Size Expertscited by 5
  9. 2026DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networkscited by 2
  10. 2026Mixture of Heterogeneous Grouped Experts for Language Modelingcited by 2
  11. 2026Tapered Language Modelscited by 2
  12. 2026Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processingcited by 2
  13. 2026Geometry-Guided Layerwise FFN Width Allocation in Transformerscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  2. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  3. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  4. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  5. 2026Towards a Statistical Understanding of Mixture-of-Experts
  6. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
  7. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  8. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models