mlsanityData to 7 Oct 2026

Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts

arXiv 2605.00604 ↗First posted 1 May 2026cs.LG cs.NEcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse MoE routing fails at domain transitions, where the current token belongs to one distribution and the next to another. In a controlled experiment (4 experts, 5 seeds), standard affinity routing assigns only 0.006 +/- 0.001 probability to the correct expert at the transition. Three lightweight gate modifications raise this to 0.748 +/- 0.002 (124x), cutting experts needed for 99% coverage from infeasible to a small constant: temporal memory (beta), a per-expert LIF membrane potential accumulating routing context across tokens; precision-weighted gating (Pi), a per-expert inverse variance of recent prediction error, yielding 31x contrast between reliable and unreliable experts; and anticipatory routing, a next-state predictor conditioned on the beta-accumulated hidden state. The mechanisms draw from Friston's Free Energy Principle and use LIF dynamics from spiking neural networks. An ablation across all 2^3 subsets reveals a super-additive beta x Ant interaction: anticipation alone gives nothing (+0.000 +/- 0.001); beta alone gives modest gain (+0.295 +/- 0.013); combined they close 75% of the oracle gap (+0.741 +/- 0.002, exceeding the sum by +0.446 +/- 0.014). This is structural: a stateless predictor cannot detect approaching transitions because pre-transition tokens are distributionally identical to within-domain tokens. In a character-level MoE LM (5 seeds), beta-routing reduces transition-step BPC from 6.56 +/- 0.01 (Standard) to 4.01 +/- 0.15 (beta-MoE); the beta + Ant gate places 0.86 +/- 0.02 probability on the correct domain expert before that domain appears in input, vs 0.42 +/- 0.12 for Standard MoE. Reference implementations (~200 lines each): https://github.com/russellwmy/affinity-is-not-enough

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 26 of its 35 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

26 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers10 Mar
  2. 2025MoxE: Mixture of xLSTM Experts with Entropy-Aware Routing for Efficient Language Modeling1 May
  3. 2024Layerwise Recurrent Router for Mixture-of-Experts13 Aug
  4. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 21

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  12. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
9 more
  1. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 251
  2. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 89
  3. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  4. 2020Deep active inference agents using Monte-Carlo methodscited by 72
  5. 2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
  6. 2022Path integrals, particular kinds, and strange thingscited by 20
  7. 2024Graph Mixture of Experts and Memory-augmented Routers for Multivariate Time Series Anomaly Detectioncited by 19
  8. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 18
  9. 2022Predictive Coding beyond Gaussian Distributionscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Expert Routing for Communication-Efficient MoE via Finite Expert Banks
  2. 2026Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
  3. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
  4. 2026Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing
  5. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inference
  6. 2026Towards a Statistical Understanding of Mixture-of-Experts
  7. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  8. 2026Cost-Penalized Fitness in FMA-Orchestrated Mixture of Experts: Experimental Evidence for Molecular Memory in Domain Adaptation