Cost-Penalized Fitness in FMA-Orchestrated Mixture of Experts: Experimental Evidence for Molecular Memory in Domain Adaptation
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
We present experimental results from seven controlled runs of nanoFMT, a Free-Market Algorithm (FMA) orchestrated transformer with dynamic Mixture-of-Experts (MoE) management. The experiments address a fundamental question for advanced LLM development: how should an MoE system manage its expert pool when operating at full capacity under changing data distributions? We demonstrate that cost-penalized fitness metrics, combined with a linear grace period for newborn experts, produce a system that accumulates domain expertise through diversification rather than replacement. The central result is a round-trip domain shift experiment showing 9-11x faster recovery when returning to a previously learned domain, with zero expert births or replacements required. This "molecular memory" effect -- where dormant experts survive and reactivate when their domain returns -- has no analogue in current MoE management approaches. A preliminary cost analysis estimates annual savings of 39.1M and 27.1 GWh energy reduction for an OpenAI-scale provider under a moderate scenario.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 6 of its 9 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
6 of 9 references.
Closest ancestors · 2
Same direction, newest first
- 2022Mixture-of-Experts with Expert Choice Routing18 Feb
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 4
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2020Ants, robots, humans: a self-organizing, complex systems modeling approachcited by 2
- 2026The Free-Market Algorithm: Self-Organizing Optimization for Open-Ended Complex Systemscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
- 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
- 2026Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
- 2026Mixture of Experts for Low-Resource LLMs
- 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
- 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
- 2026It Takes a MAESTRO To Prune Bad Experts
- 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation