mlsanityData to 7 Oct 2026

Neural Inhibition Improves Dynamic Routing and Mixture of Experts

arXiv 2507.03221 ↗First posted 3 Jul 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

To be effective, efficient, and diverse, deep learning models need to dynamically choose its architecture based on signals from a population of neurons. We hypothesize dynamic routing models can be improved with neural inhibition in those neural populations. This means signals commonly shared among the various modes of data statistics can be inhibited so that the routing model can choose a specialized expert path for each data sample. Only through inhibition is the routing mechanism able to effectively select neural pathways. We believe this is an under-studied and under-verified implementation methodology for Mixture-of-Experts, dynamic routing, and transformer language models. We provide experimental evidence that the neural inhibition algorithm significantly boosts the performance of general tasks and motivates more effort to be invested in this research direction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 19 of its 41 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

19 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion5 Feb
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 15

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  6. 2016Language Modeling with Gated Convolutional Networkscited by 688
  7. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  8. 2018DropBlock: A regularization method for convolutional networkscited by 261
  9. 2015Highway Networkscited by 257
  10. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  11. 2019Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networkscited by 103
  12. 2018Channel Gating Neural Networkscited by 80
3 more
  1. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  2. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  3. 2024Efficient Sparse Coding with the Adaptive Locally Competitive Algorithm for Speech Classificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  2. 2026SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts
  3. 2026Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
  4. 2023Soft Merging of Experts with Adaptive Routing
  5. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
  6. 2026Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
  7. 2026Path-Constrained Mixture-of-Experts
  8. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers