mlsanityData to 7 Oct 2026

Excitation: Momentum For Experts

arXiv 2602.21798 ↗First posted 25 Feb 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

We propose Excitation, a novel optimization framework designed to accelerate learning in sparse architectures such as Mixture-of-Experts (MoEs). Unlike traditional optimizers that treat all parameters uniformly, Excitation dynamically modulates updates using batch-level expert utilization. It introduces a competitive update dynamic that amplifies updates to highly-utilized experts and can selectively suppress low-utilization ones, effectively sharpening routing specialization. Notably, we identify a phenomenon of "structural confusion" in deep MoEs, where standard optimizers fail to establish functional signal paths; Excitation acts as a specialization catalyst, "rescuing" these models and enabling stable training where baselines remain trapped. Excitation is optimizer-, domain-, and model-agnostic, requires minimal integration effort, and introduces neither additional per-parameter optimizer state nor learnable parameters, making it highly viable for memory-constrained settings. Across language and vision tasks, Excitation consistently improves convergence speed and final performance in MoE models, indicating that active update modulation is a key mechanism for effective conditional computation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 29 of its 33 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

29 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MoxE: Mixture of xLSTM Experts with Entropy-Aware Routing for Efficient Language Modeling1 May
  2. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  3. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts12 Dec
  4. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  5. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2017Large Batch Training of Convolutional Networkscited by 529
  10. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  11. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  12. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
11 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  2. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  4. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  5. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  6. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  7. 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
  8. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  9. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  10. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  11. 2024More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoE
  2. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
  3. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  4. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  5. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
  6. 2026Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
  7. 2026Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees
  8. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices