mlsanityData to 7 Oct 2026

Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing

arXiv 2603.11535 ↗First posted 12 Mar 2026cs.AI cs.CLcited by 3

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and requiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving average (EMA) threshold estimated from the global token distribution. At both training and inference, each token is independently routed to an expert if its score exceeds the expert's threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates dependence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B parameters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6× fewer tokens.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 34 of its 52 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

34 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  2. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  3. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models19 Jun
  4. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models23 May
  5. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models12 Mar
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
16 more
  1. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  2. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  3. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  4. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  5. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  6. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  7. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  8. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  9. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  10. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  11. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  12. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  13. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  14. 2025UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoEcited by 4
  15. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 2
  16. 2026Improving MoE Compute Efficiency by Composing Weight and Data Sparsitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Kimi K3: Open Frontier Intelligencecited by 99
  2. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  3. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  2. 2026SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
  3. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
  4. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
  5. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  6. 2026Mixture of Experts for Low-Resource LLMs
  7. 2025Route Experts by Sequence, not by Token
  8. 2026It Takes a MAESTRO To Prune Bad Experts